Jul 07 06:29:48.531577 ip-10-0-133-92 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 07 06:29:48.531589 ip-10-0-133-92 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 07 06:29:48.531599 ip-10-0-133-92 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 07 06:29:48.531933 ip-10-0-133-92 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 07 06:29:58.618825 ip-10-0-133-92 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 07 06:29:58.618839 ip-10-0-133-92 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 84edc61546a24824a52e4755746c5404 --
Jul 07 06:31:54.858196 ip-10-0-133-92 systemd[1]: Starting Kubernetes Kubelet...
Jul 07 06:31:55.271314 ip-10-0-133-92 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 06:31:55.271314 ip-10-0-133-92 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 07 06:31:55.271314 ip-10-0-133-92 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 06:31:55.271314 ip-10-0-133-92 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 07 06:31:55.271314 ip-10-0-133-92 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 06:31:55.272040 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.271318    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 07 06:31:55.277510 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277482    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 06:31:55.277510 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277506    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 06:31:55.277510 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277510    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 06:31:55.277510 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277513    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 06:31:55.277510 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277517    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 06:31:55.277742 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277523    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 06:31:55.277742 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277527    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 06:31:55.277742 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277531    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 06:31:55.277742 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277535    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 06:31:55.277742 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277539    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 06:31:55.277742 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277541    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 06:31:55.277742 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277544    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 06:31:55.277742 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277547    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 06:31:55.277742 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277550    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 06:31:55.277742 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277552    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 06:31:55.277742 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277555    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 06:31:55.277742 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277557    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 06:31:55.277742 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277559    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 06:31:55.277742 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277562    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 06:31:55.277742 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277564    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 06:31:55.277742 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277567    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 06:31:55.277742 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277569    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 06:31:55.277742 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277572    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 06:31:55.277742 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277574    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 06:31:55.278255 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277577    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 06:31:55.278255 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277579    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 06:31:55.278255 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277582    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 06:31:55.278255 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277594    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 06:31:55.278255 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277597    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 06:31:55.278255 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277599    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 06:31:55.278255 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277602    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 06:31:55.278255 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277604    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 06:31:55.278255 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277607    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 06:31:55.278255 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277610    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 06:31:55.278255 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277612    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 06:31:55.278255 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277616    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 06:31:55.278255 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277619    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 06:31:55.278255 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277622    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 06:31:55.278255 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277626    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 06:31:55.278255 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277628    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 06:31:55.278255 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277631    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 06:31:55.278255 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277634    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 06:31:55.278255 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277637    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 06:31:55.278716 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277640    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 06:31:55.278716 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277642    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 06:31:55.278716 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277645    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 06:31:55.278716 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277648    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 06:31:55.278716 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277650    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 06:31:55.278716 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277652    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 06:31:55.278716 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277655    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 06:31:55.278716 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277658    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 06:31:55.278716 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277660    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 06:31:55.278716 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277662    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 06:31:55.278716 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277665    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 06:31:55.278716 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277668    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 06:31:55.278716 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277670    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 06:31:55.278716 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277672    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 06:31:55.278716 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277675    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 06:31:55.278716 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277678    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 06:31:55.278716 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277680    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 06:31:55.278716 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277683    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 06:31:55.278716 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277685    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 06:31:55.278716 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277689    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 06:31:55.279218 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277691    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 06:31:55.279218 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277694    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 06:31:55.279218 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277696    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 06:31:55.279218 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277698    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 06:31:55.279218 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277701    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 06:31:55.279218 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277704    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 06:31:55.279218 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277707    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 06:31:55.279218 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277710    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 06:31:55.279218 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277712    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 06:31:55.279218 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277715    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 06:31:55.279218 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277718    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 06:31:55.279218 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277721    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 06:31:55.279218 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277723    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 06:31:55.279218 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277726    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 06:31:55.279218 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277729    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 06:31:55.279218 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277732    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 06:31:55.279218 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277734    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 06:31:55.279218 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277737    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 06:31:55.279218 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277739    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 06:31:55.279218 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277742    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 06:31:55.279712 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277745    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 06:31:55.279712 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277748    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 06:31:55.279712 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277750    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 06:31:55.279712 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277753    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 06:31:55.279712 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277755    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 06:31:55.279712 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277758    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 06:31:55.279712 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277761    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 06:31:55.279712 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277763    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 06:31:55.279712 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277765    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 06:31:55.279712 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277768    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 06:31:55.279712 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277770    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 06:31:55.279712 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277773    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 06:31:55.279712 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277775    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 06:31:55.279712 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277778    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 06:31:55.279712 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277780    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 06:31:55.279712 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.277782    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 06:31:55.279712 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279054    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 06:31:55.279712 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279063    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 06:31:55.279712 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279066    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 06:31:55.279712 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279071    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 06:31:55.280214 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279074    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 06:31:55.280214 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279077    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 06:31:55.280214 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279080    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 06:31:55.280214 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279082    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 06:31:55.280214 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279085    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 06:31:55.280214 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279087    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 06:31:55.280214 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279090    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 06:31:55.280214 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279092    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 06:31:55.280214 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279095    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 06:31:55.280214 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279097    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 06:31:55.280214 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279100    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 06:31:55.280214 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279102    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 06:31:55.280214 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279106    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 06:31:55.280214 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279111    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 06:31:55.280214 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279115    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 06:31:55.280214 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279118    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 06:31:55.280214 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279120    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 06:31:55.280214 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279123    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 06:31:55.280214 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279126    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 06:31:55.280685 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279128    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 06:31:55.280685 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279131    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 06:31:55.280685 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279134    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 06:31:55.280685 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279137    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 06:31:55.280685 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279140    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 06:31:55.280685 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279142    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 06:31:55.280685 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279144    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 06:31:55.280685 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279147    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 06:31:55.280685 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279149    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 06:31:55.280685 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279152    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 06:31:55.280685 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279155    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 06:31:55.280685 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279157    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 06:31:55.280685 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279160    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 06:31:55.280685 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279163    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 06:31:55.280685 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279165    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 06:31:55.280685 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279168    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 06:31:55.280685 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279171    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 06:31:55.280685 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279174    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 06:31:55.280685 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279177    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 06:31:55.280685 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279179    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 06:31:55.280685 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279182    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 06:31:55.281222 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279184    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 06:31:55.281222 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279187    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 06:31:55.281222 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279189    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 06:31:55.281222 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279192    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 06:31:55.281222 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279195    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 06:31:55.281222 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279197    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 06:31:55.281222 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279200    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 06:31:55.281222 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279203    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 06:31:55.281222 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279206    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 06:31:55.281222 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279208    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 06:31:55.281222 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279211    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 06:31:55.281222 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279213    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 06:31:55.281222 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279216    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 06:31:55.281222 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279219    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 06:31:55.281222 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279222    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 06:31:55.281222 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279225    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 06:31:55.281222 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279228    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 06:31:55.281222 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279232    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 06:31:55.281222 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279236    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 06:31:55.281691 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279239    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 06:31:55.281691 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279241    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 06:31:55.281691 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279243    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 06:31:55.281691 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279247    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 06:31:55.281691 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279250    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 06:31:55.281691 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279253    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 06:31:55.281691 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279255    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 06:31:55.281691 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279258    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 06:31:55.281691 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279260    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 06:31:55.281691 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279263    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 06:31:55.281691 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279265    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 06:31:55.281691 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279268    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 06:31:55.281691 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279270    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 06:31:55.281691 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279273    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 06:31:55.281691 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279276    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 06:31:55.281691 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279278    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 06:31:55.281691 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279280    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 06:31:55.281691 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279282    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 06:31:55.281691 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279285    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 06:31:55.281691 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279287    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 06:31:55.282226 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279290    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 06:31:55.282226 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279293    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 06:31:55.282226 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279295    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 06:31:55.282226 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279297    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 06:31:55.282226 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279300    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 06:31:55.282226 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279302    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 06:31:55.282226 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279305    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 06:31:55.282226 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279308    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 06:31:55.282226 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279310    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 06:31:55.282226 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279313    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 06:31:55.282226 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279315    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 06:31:55.282226 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279317    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 06:31:55.282226 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279320    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 06:31:55.282226 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279323    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 06:31:55.282226 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279326    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 06:31:55.282226 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279329    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 06:31:55.282226 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279742    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 06:31:55.282226 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279747    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 06:31:55.282226 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279750    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 06:31:55.282682 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279753    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 06:31:55.282682 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279755    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 06:31:55.282682 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279758    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 06:31:55.282682 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279761    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 06:31:55.282682 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279763    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 06:31:55.282682 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279765    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 06:31:55.282682 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279768    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 06:31:55.282682 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279771    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 06:31:55.282682 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279773    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 06:31:55.282682 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279776    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 06:31:55.282682 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279778    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 06:31:55.282682 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279781    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 06:31:55.282682 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279783    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 06:31:55.282682 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279787    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 06:31:55.282682 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279789    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 06:31:55.282682 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279792    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 06:31:55.282682 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279794    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 06:31:55.282682 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279796    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 06:31:55.282682 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279799    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 06:31:55.282682 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279802    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 06:31:55.283235 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279805    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 06:31:55.283235 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279807    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 06:31:55.283235 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279809    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 06:31:55.283235 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279812    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 06:31:55.283235 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279814    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 06:31:55.283235 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279816    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 06:31:55.283235 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279833    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 06:31:55.283235 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279836    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 06:31:55.283235 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279838    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 06:31:55.283235 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279842    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 06:31:55.283235 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279845    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 06:31:55.283235 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279847    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 06:31:55.283235 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279850    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 06:31:55.283235 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279852    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 06:31:55.283235 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279854    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 06:31:55.283235 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279857    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 06:31:55.283235 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279860    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 06:31:55.283235 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279862    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 06:31:55.283235 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279865    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 06:31:55.283235 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279867    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 06:31:55.283806 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279870    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 06:31:55.283806 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279872    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 06:31:55.283806 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279875    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 06:31:55.283806 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279877    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 06:31:55.283806 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279880    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 06:31:55.283806 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279882    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 06:31:55.283806 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279885    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 06:31:55.283806 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279887    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 06:31:55.283806 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279890    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 06:31:55.283806 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279892    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 06:31:55.283806 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279895    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 06:31:55.283806 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279898    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 06:31:55.283806 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279901    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 06:31:55.283806 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279903    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 06:31:55.283806 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279906    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 06:31:55.283806 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279908    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 06:31:55.283806 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279910    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 06:31:55.283806 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279912    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 06:31:55.283806 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279915    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 06:31:55.284292 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279919    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 06:31:55.284292 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279922    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 06:31:55.284292 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279925    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 06:31:55.284292 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279928    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 06:31:55.284292 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279931    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 06:31:55.284292 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279934    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 06:31:55.284292 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279937    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 06:31:55.284292 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279939    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 06:31:55.284292 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279941    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 06:31:55.284292 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279944    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 06:31:55.284292 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279946    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 06:31:55.284292 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279949    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 06:31:55.284292 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279951    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 06:31:55.284292 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279953    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 06:31:55.284292 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279955    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 06:31:55.284292 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279958    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 06:31:55.284292 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279960    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 06:31:55.284292 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279965    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 06:31:55.284758 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279968    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 06:31:55.284758 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279972    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 06:31:55.284758 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279975    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 06:31:55.284758 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279978    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 06:31:55.284758 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279980    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 06:31:55.284758 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279982    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 06:31:55.284758 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279985    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 06:31:55.284758 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279987    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 06:31:55.284758 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279990    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 06:31:55.284758 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279992    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 06:31:55.284758 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279994    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 06:31:55.284758 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279997    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 06:31:55.284758 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.279999    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 06:31:55.284758 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280001    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 06:31:55.284758 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280004    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 06:31:55.284758 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280006    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 06:31:55.284758 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280009    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 06:31:55.284758 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280012    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 06:31:55.284758 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280014    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 06:31:55.284758 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280086    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 07 06:31:55.285251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280094    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 07 06:31:55.285251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280102    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 07 06:31:55.285251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280108    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 07 06:31:55.285251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280115    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 07 06:31:55.285251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280120    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 07 06:31:55.285251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280127    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 07 06:31:55.285251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280131    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 07 06:31:55.285251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280134    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 07 06:31:55.285251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280137    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 07 06:31:55.285251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280141    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 07 06:31:55.285251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280144    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 07 06:31:55.285251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280147    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 07 06:31:55.285251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280150    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 07 06:31:55.285251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280153    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 07 06:31:55.285251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280156    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 07 06:31:55.285251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280161    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 07 06:31:55.285251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280164    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 07 06:31:55.285251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280177    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 07 06:31:55.285251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280180    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 07 06:31:55.285251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280183    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 07 06:31:55.285251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280186    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 07 06:31:55.285251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280189    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 07 06:31:55.285251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280193    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 07 06:31:55.285251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280196    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 07 06:31:55.285864 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280199    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 07 06:31:55.285864 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280202    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 07 06:31:55.285864 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280206    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 07 06:31:55.285864 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280208    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 07 06:31:55.285864 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280211    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 07 06:31:55.285864 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280214    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 07 06:31:55.285864 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280217    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 07 06:31:55.285864 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280221    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 07 06:31:55.285864 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280224    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 07 06:31:55.285864 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280227    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 07 06:31:55.285864 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280230    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 07 06:31:55.285864 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280234    2563 flags.go:64] FLAG: --enable-server="true"
Jul 07 06:31:55.285864 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280237    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 07 06:31:55.285864 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280242    2563 flags.go:64] FLAG: --event-burst="100"
Jul 07 06:31:55.285864 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280245    2563 flags.go:64] FLAG: --event-qps="50"
Jul 07 06:31:55.285864 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280247    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 07 06:31:55.285864 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280250    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 07 06:31:55.285864 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280253    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 07 06:31:55.285864 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280256    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 07 06:31:55.285864 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280259    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 07 06:31:55.285864 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280262    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 07 06:31:55.285864 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280265    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 07 06:31:55.285864 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280267    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 07 06:31:55.285864 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280270    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 07 06:31:55.285864 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280277    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 07 06:31:55.286484 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280279    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 07 06:31:55.286484 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280282    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 07 06:31:55.286484 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280285    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 07 06:31:55.286484 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280288    2563 flags.go:64] FLAG: --feature-gates=""
Jul 07 06:31:55.286484 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280291    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 07 06:31:55.286484 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280294    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 07 06:31:55.286484 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280297    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 07 06:31:55.286484 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280300    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 07 06:31:55.286484 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280302    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 07 06:31:55.286484 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280305    2563 flags.go:64] FLAG: --help="false"
Jul 07 06:31:55.286484 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280308    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-133-92.ec2.internal"
Jul 07 06:31:55.286484 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280313    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 07 06:31:55.286484 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280316    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 07 06:31:55.286484 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280319    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 07 06:31:55.286484 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280322    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 07 06:31:55.286484 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280325    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 07 06:31:55.286484 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280328    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 07 06:31:55.286484 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280331    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 07 06:31:55.286484 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280334    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 07 06:31:55.286484 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280336    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 07 06:31:55.286484 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280339    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 07 06:31:55.286484 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280342    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 07 06:31:55.286484 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280344    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 07 06:31:55.286484 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280347    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 07 06:31:55.287103 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280350    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 07 06:31:55.287103 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280353    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 07 06:31:55.287103 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280355    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 07 06:31:55.287103 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280358    2563 flags.go:64] FLAG: --lock-file=""
Jul 07 06:31:55.287103 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280361    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 07 06:31:55.287103 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280364    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 07 06:31:55.287103 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280366    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 07 06:31:55.287103 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280372    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 07 06:31:55.287103 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280375    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 07 06:31:55.287103 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280381    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 07 06:31:55.287103 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280384    2563 flags.go:64] FLAG: --logging-format="text"
Jul 07 06:31:55.287103 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280386    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 07 06:31:55.287103 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280390    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 07 06:31:55.287103 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280392    2563 flags.go:64] FLAG: --manifest-url=""
Jul 07 06:31:55.287103 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280395    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 07 06:31:55.287103 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280399    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 07 06:31:55.287103 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280402    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 07 06:31:55.287103 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280406    2563 flags.go:64] FLAG: --max-pods="110"
Jul 07 06:31:55.287103 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280409    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 07 06:31:55.287103 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280413    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 07 06:31:55.287103 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280415    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 07 06:31:55.287103 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280418    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 07 06:31:55.287103 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280421    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 07 06:31:55.287103 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280424    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 07 06:31:55.287103 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280427    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 07 06:31:55.288076 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280434    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 07 06:31:55.288076 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280437    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 07 06:31:55.288076 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280440    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 07 06:31:55.288076 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280443    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 07 06:31:55.288076 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280445    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 07 06:31:55.288076 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280451    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 07 06:31:55.288076 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280454    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 07 06:31:55.288076 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280457    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 07 06:31:55.288076 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280460    2563 flags.go:64] FLAG: --port="10250"
Jul 07 06:31:55.288076 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280463    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 07 06:31:55.288076 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280466    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08827001039e5fb02"
Jul 07 06:31:55.288076 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280469    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 07 06:31:55.288076 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280471    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 07 06:31:55.288076 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280474    2563 flags.go:64] FLAG: --register-node="true"
Jul 07 06:31:55.288076 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280477    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 07 06:31:55.288076 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280480    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 07 06:31:55.288076 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280487    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 07 06:31:55.288076 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280490    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 07 06:31:55.288076 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280492    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 07 06:31:55.288076 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280496    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 07 06:31:55.288076 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280499    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 07 06:31:55.288076 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280501    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 07 06:31:55.288076 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280504    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 07 06:31:55.288076 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280507    2563 flags.go:64] FLAG: --runonce="false"
Jul 07 06:31:55.288076 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280509    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 07 06:31:55.289223 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280512    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 07 06:31:55.289223 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280515    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 07 06:31:55.289223 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280519    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 07 06:31:55.289223 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280522    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 07 06:31:55.289223 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280525    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 07 06:31:55.289223 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280528    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 07 06:31:55.289223 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280531    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 07 06:31:55.289223 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280534    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 07 06:31:55.289223 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280537    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 07 06:31:55.289223 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280539    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 07 06:31:55.289223 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280542    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 07 06:31:55.289223 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280545    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 07 06:31:55.289223 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280547    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 07 06:31:55.289223 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280550    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 07 06:31:55.289223 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280555    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 07 06:31:55.289223 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280558    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 07 06:31:55.289223 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280560    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 07 06:31:55.289223 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280564    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 07 06:31:55.289223 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280567    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 07 06:31:55.289223 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280569    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 07 06:31:55.289223 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280572    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 07 06:31:55.289223 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280575    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 07 06:31:55.289223 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280578    2563 flags.go:64] FLAG: --v="2"
Jul 07 06:31:55.289223 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280584    2563 flags.go:64] FLAG: --version="false"
Jul 07 06:31:55.289223 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280588    2563 flags.go:64] FLAG: --vmodule=""
Jul 07 06:31:55.290302 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280592    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 07 06:31:55.290302 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.280595    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 07 06:31:55.290302 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280725    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 06:31:55.290302 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280729    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 06:31:55.290302 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280732    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 06:31:55.290302 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280734    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 06:31:55.290302 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280737    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 06:31:55.290302 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280739    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 06:31:55.290302 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280742    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 06:31:55.290302 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280746    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 06:31:55.290302 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280748    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 06:31:55.290302 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280751    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 06:31:55.290302 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280754    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 06:31:55.290302 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280756    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 06:31:55.290302 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280758    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 06:31:55.290302 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280761    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 06:31:55.290302 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280763    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 06:31:55.290302 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280766    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 06:31:55.290302 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280769    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 06:31:55.290302 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280771    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 06:31:55.291337 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280774    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 06:31:55.291337 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280776    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 06:31:55.291337 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280779    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 06:31:55.291337 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280781    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 06:31:55.291337 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280783    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 06:31:55.291337 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280786    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 06:31:55.291337 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280788    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 06:31:55.291337 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280791    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 06:31:55.291337 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280793    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 06:31:55.291337 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280796    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 06:31:55.291337 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280800    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 06:31:55.291337 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280802    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 06:31:55.291337 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280810    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 06:31:55.291337 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280813    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 06:31:55.291337 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280815    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 06:31:55.291337 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280818    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 06:31:55.291337 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280834    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 06:31:55.291337 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280836    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 06:31:55.291337 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280839    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 06:31:55.291337 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280841    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 06:31:55.292040 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280843    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 06:31:55.292040 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280849    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 06:31:55.292040 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280851    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 06:31:55.292040 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280854    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 06:31:55.292040 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280856    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 06:31:55.292040 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280859    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 06:31:55.292040 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280861    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 06:31:55.292040 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280863    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 06:31:55.292040 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280866    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 06:31:55.292040 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280868    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 06:31:55.292040 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280870    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 06:31:55.292040 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280873    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 06:31:55.292040 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280875    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 06:31:55.292040 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280879    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 06:31:55.292040 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280883    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 06:31:55.292040 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280886    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 06:31:55.292040 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280889    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 06:31:55.292040 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280891    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 06:31:55.292040 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280894    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 06:31:55.292040 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280896    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 06:31:55.292771 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280898    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 06:31:55.292771 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280900    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 06:31:55.292771 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280904    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 06:31:55.292771 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280906    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 06:31:55.292771 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280909    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 06:31:55.292771 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280917    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 06:31:55.292771 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280920    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 06:31:55.292771 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280922    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 06:31:55.292771 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280925    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 06:31:55.292771 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280928    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 06:31:55.292771 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280930    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 06:31:55.292771 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280933    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 06:31:55.292771 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280935    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 06:31:55.292771 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280940    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 06:31:55.292771 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280942    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 06:31:55.292771 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280945    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 06:31:55.292771 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280947    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 06:31:55.292771 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280949    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 06:31:55.292771 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280952    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 06:31:55.292771 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280954    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 06:31:55.293618 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280957    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 06:31:55.293618 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280959    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 06:31:55.293618 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280962    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 06:31:55.293618 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280964    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 06:31:55.293618 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280966    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 06:31:55.293618 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280969    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 06:31:55.293618 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280971    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 06:31:55.293618 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280974    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 06:31:55.293618 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280976    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 06:31:55.293618 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280979    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 06:31:55.293618 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280982    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 06:31:55.293618 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280986    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 06:31:55.293618 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280989    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 06:31:55.293618 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280992    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 06:31:55.293618 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280995    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 06:31:55.293618 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.280998    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 06:31:55.293618 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.281001    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 06:31:55.293618 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.281004    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 06:31:55.293618 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.281015    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 06:31:55.294221 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.281018    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 06:31:55.294221 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.281021    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 06:31:55.294221 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.281993    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 06:31:55.294221 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.290006    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 07 06:31:55.294221 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.290034    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 07 06:31:55.294221 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.290063    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 07 06:31:55.294221 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.290073    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 07 06:31:55.294221 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290192    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 06:31:55.294221 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290201    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 06:31:55.294221 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290209    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 06:31:55.294221 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290215    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 06:31:55.294221 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290220    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 06:31:55.294221 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290224    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 06:31:55.294221 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290228    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 06:31:55.294672 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290233    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 06:31:55.294672 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290237    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 06:31:55.294672 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290241    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 06:31:55.294672 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290245    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 06:31:55.294672 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290249    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 06:31:55.294672 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290253    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 06:31:55.294672 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290257    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 06:31:55.294672 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290261    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 06:31:55.294672 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290266    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 06:31:55.294672 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290270    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 06:31:55.294672 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290274    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 06:31:55.294672 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290278    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 06:31:55.294672 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290285    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 06:31:55.294672 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290291    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 06:31:55.294672 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290296    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 06:31:55.294672 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290300    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 06:31:55.294672 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290304    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 06:31:55.294672 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290308    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 06:31:55.294672 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290312    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 06:31:55.295241 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290315    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 06:31:55.295241 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290319    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 06:31:55.295241 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290323    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 06:31:55.295241 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290337    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 06:31:55.295241 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290342    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 06:31:55.295241 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290346    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 06:31:55.295241 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290351    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 06:31:55.295241 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290355    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 06:31:55.295241 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290359    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 06:31:55.295241 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290363    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 06:31:55.295241 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290368    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 06:31:55.295241 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290372    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 06:31:55.295241 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290376    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 06:31:55.295241 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290381    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 06:31:55.295241 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290385    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 06:31:55.295241 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290389    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 06:31:55.295241 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290393    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 06:31:55.295241 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290398    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 06:31:55.295241 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290402    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 06:31:55.295241 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290406    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 06:31:55.295726 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290410    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 06:31:55.295726 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290415    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 06:31:55.295726 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290419    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 06:31:55.295726 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290423    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 06:31:55.295726 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290427    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 06:31:55.295726 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290431    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 06:31:55.295726 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290436    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 06:31:55.295726 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290440    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 06:31:55.295726 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290444    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 06:31:55.295726 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290449    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 06:31:55.295726 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290453    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 06:31:55.295726 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290457    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 06:31:55.295726 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290461    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 06:31:55.295726 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290465    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 06:31:55.295726 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290469    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 06:31:55.295726 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290473    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 06:31:55.295726 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290485    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 06:31:55.295726 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290491    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 06:31:55.295726 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290495    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 06:31:55.295726 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290499    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 06:31:55.296323 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290503    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 06:31:55.296323 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290507    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 06:31:55.296323 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290511    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 06:31:55.296323 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290515    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 06:31:55.296323 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290520    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 06:31:55.296323 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290524    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 06:31:55.296323 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290529    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 06:31:55.296323 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290533    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 06:31:55.296323 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290537    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 06:31:55.296323 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290541    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 06:31:55.296323 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290547    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 06:31:55.296323 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290552    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 06:31:55.296323 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290556    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 06:31:55.296323 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290560    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 06:31:55.296323 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290565    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 06:31:55.296323 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290569    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 06:31:55.296323 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290573    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 06:31:55.296323 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290577    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 06:31:55.296323 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290581    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 06:31:55.296802 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290586    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 06:31:55.296802 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290590    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 06:31:55.296802 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290594    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 06:31:55.296802 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290599    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 06:31:55.296802 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290603    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 06:31:55.296802 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290608    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 06:31:55.296802 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290612    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 06:31:55.296802 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290616    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 06:31:55.296802 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290621    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 06:31:55.296802 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290625    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 06:31:55.296802 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290638    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 06:31:55.296802 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290644    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 06:31:55.296802 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290648    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 06:31:55.296802 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290652    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 06:31:55.296802 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.290660    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 06:31:55.297254 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290907    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 06:31:55.297254 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290919    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 06:31:55.297254 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290925    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 06:31:55.297254 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290929    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 06:31:55.297254 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290933    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 06:31:55.297254 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290937    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 06:31:55.297254 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290940    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 06:31:55.297254 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290944    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 06:31:55.297254 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290947    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 06:31:55.297254 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290951    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 06:31:55.297254 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290954    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 06:31:55.297254 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290958    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 06:31:55.297254 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290962    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 06:31:55.297254 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290966    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 06:31:55.297254 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290970    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 06:31:55.297254 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290973    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 06:31:55.297254 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290978    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 06:31:55.297254 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290982    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 06:31:55.297254 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290986    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 06:31:55.297254 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290990    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 06:31:55.297768 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290993    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 06:31:55.297768 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.290998    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 06:31:55.297768 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291002    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 06:31:55.297768 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291006    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 06:31:55.297768 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291010    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 06:31:55.297768 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291015    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 06:31:55.297768 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291019    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 06:31:55.297768 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291023    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 06:31:55.297768 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291035    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 06:31:55.297768 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291040    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 06:31:55.297768 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291044    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 06:31:55.297768 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291048    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 06:31:55.297768 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291052    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 06:31:55.297768 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291058    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 06:31:55.297768 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291064    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 06:31:55.297768 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291068    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 06:31:55.297768 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291073    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 06:31:55.297768 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291077    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 06:31:55.297768 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291082    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 06:31:55.297768 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291087    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 06:31:55.298334 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291091    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 06:31:55.298334 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291096    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 06:31:55.298334 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291100    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 06:31:55.298334 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291104    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 06:31:55.298334 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291109    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 06:31:55.298334 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291113    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 06:31:55.298334 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291117    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 06:31:55.298334 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291121    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 06:31:55.298334 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291126    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 06:31:55.298334 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291131    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 06:31:55.298334 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291135    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 06:31:55.298334 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291139    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 06:31:55.298334 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291144    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 06:31:55.298334 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291148    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 06:31:55.298334 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291152    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 06:31:55.298334 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291156    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 06:31:55.298334 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291160    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 06:31:55.298334 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291164    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 06:31:55.298334 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291168    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 06:31:55.299142 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291172    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 06:31:55.299142 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291176    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 06:31:55.299142 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291189    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 06:31:55.299142 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291193    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 06:31:55.299142 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291197    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 06:31:55.299142 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291201    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 06:31:55.299142 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291205    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 06:31:55.299142 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291209    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 06:31:55.299142 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291213    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 06:31:55.299142 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291218    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 06:31:55.299142 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291222    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 06:31:55.299142 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291226    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 06:31:55.299142 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291230    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 06:31:55.299142 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291234    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 06:31:55.299142 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291238    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 06:31:55.299142 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291243    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 06:31:55.299142 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291247    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 06:31:55.299142 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291251    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 06:31:55.299142 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291255    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 06:31:55.299931 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291259    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 06:31:55.299931 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291264    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 06:31:55.299931 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291269    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 06:31:55.299931 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291274    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 06:31:55.299931 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291278    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 06:31:55.299931 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291282    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 06:31:55.299931 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291287    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 06:31:55.299931 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291291    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 06:31:55.299931 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291295    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 06:31:55.299931 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291299    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 06:31:55.299931 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291303    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 06:31:55.299931 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291307    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 06:31:55.299931 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291311    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 06:31:55.299931 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291315    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 06:31:55.299931 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291319    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 06:31:55.299931 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291323    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 06:31:55.299931 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291335    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 06:31:55.299931 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291340    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 06:31:55.299931 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291343    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 06:31:55.299931 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291347    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 06:31:55.300443 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:55.291351    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 06:31:55.300443 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.291358    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 06:31:55.300443 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.291607    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 07 06:31:55.300443 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.294509    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 07 06:31:55.300443 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.295974    2563 server.go:1019] "Starting client certificate rotation"
Jul 07 06:31:55.300443 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.296078    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 06:31:55.300443 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.296946    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 06:31:55.300443 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.299058    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 06:31:55.304099 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.304073    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 06:31:55.319550 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.319529    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 07 06:31:55.321521 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.321505    2563 log.go:25] "Validated CRI v1 image API"
Jul 07 06:31:55.322378 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.322360    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 07 06:31:55.324956 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.324929    2563 fs.go:135] Filesystem UUIDs: map[0c60eaaf-fd92-49f5-9228-e059604702f0:/dev/nvme0n1p4 67090f3d-6534-4af5-b881-0137669d90c4:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 07 06:31:55.325033 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.324958    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 07 06:31:55.325033 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.324936    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 06:31:55.331229 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.331105    2563 manager.go:216] Machine: {Timestamp:2026-07-07 06:31:55.329418521 +0000 UTC m=+0.355427913 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3090281 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2680560e8b05dc1ffe9cbf52719d9a SystemUUID:ec268056-0e8b-05dc-1ffe-9cbf52719d9a BootID:84edc615-46a2-4824-a52e-4755746c5404 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3d:ed:d4:41:df Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3d:ed:d4:41:df Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:c3:2e:7d:31:45 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 07 06:31:55.331229 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.331224    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 07 06:31:55.331332 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.331310    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 07 06:31:55.334575 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.334539    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 07 06:31:55.334731 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.334579    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-92.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 07 06:31:55.336244 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.336232    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 07 06:31:55.336285 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.336247    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 07 06:31:55.336285 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.336255    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 06:31:55.336905 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.336874    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 06:31:55.338153 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.338140    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 07 06:31:55.340995 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.340982    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 06:31:55.341128 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.341118    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 07 06:31:55.341159 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.341150    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 07 06:31:55.341186 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.341161    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 07 06:31:55.341186 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.341172    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 07 06:31:55.341251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.341186    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 07 06:31:55.341251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.341196    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 07 06:31:55.341603 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.341592    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 06:31:55.341653 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.341614    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 06:31:55.343872 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.343854    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 07 06:31:55.345590 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.345573    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 07 06:31:55.345650 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.345607    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 07 06:31:55.347055 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.347042    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 07 06:31:55.347055 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.347057    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 07 06:31:55.347157 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.347063    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 07 06:31:55.347157 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.347068    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 07 06:31:55.347157 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.347077    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 07 06:31:55.347157 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.347082    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 07 06:31:55.347157 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.347073    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fpc2p"
Jul 07 06:31:55.347157 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.347092    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 07 06:31:55.347157 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.347100    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 07 06:31:55.347157 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.347110    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 07 06:31:55.347157 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.347118    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 07 06:31:55.347157 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.347127    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 07 06:31:55.347157 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.347132    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 07 06:31:55.347893 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.347882    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 07 06:31:55.347893 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.347893    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 07 06:31:55.351401 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.351386    2563 server.go:1268] "Started kubelet"
Jul 07 06:31:55.352145 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.352115    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 07 06:31:55.352279 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.352162    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 07 06:31:55.352476 ip-10-0-133-92 systemd[1]: Started Kubernetes Kubelet.
Jul 07 06:31:55.352707 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.352617    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 07 06:31:55.354895 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.354032    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 07 06:31:55.356131 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.356112    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fpc2p"
Jul 07 06:31:55.356351 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.356334    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 07 06:31:55.356978 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.356948    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-92.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 07 06:31:55.357056 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:55.357008    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-92.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 07 06:31:55.357217 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:55.357186    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 07 06:31:55.358689 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:55.358662    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 07 06:31:55.361941 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.361914    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 07 06:31:55.361941 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.361932    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 07 06:31:55.362098 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:55.362043    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-92.ec2.internal\" not found"
Jul 07 06:31:55.362586 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:55.361274    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-92.ec2.internal.18bfee3b2ad90bf1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-92.ec2.internal,UID:ip-10-0-133-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-92.ec2.internal,},FirstTimestamp:2026-07-07 06:31:55.351354353 +0000 UTC m=+0.377363730,LastTimestamp:2026-07-07 06:31:55.351354353 +0000 UTC m=+0.377363730,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-92.ec2.internal,}"
Jul 07 06:31:55.362726 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.362646    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 07 06:31:55.362788 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.362710    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 07 06:31:55.362788 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.362743    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 07 06:31:55.362918 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.362900    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 07 06:31:55.362918 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.362917    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 07 06:31:55.363498 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.363368    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 07 06:31:55.363498 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.363385    2563 factory.go:55] Registering systemd factory
Jul 07 06:31:55.363498 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.363395    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 07 06:31:55.363751 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.363677    2563 factory.go:153] Registering CRI-O factory
Jul 07 06:31:55.363751 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.363700    2563 factory.go:223] Registration of the crio container factory successfully
Jul 07 06:31:55.363751 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.363730    2563 factory.go:103] Registering Raw factory
Jul 07 06:31:55.363751 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.363746    2563 manager.go:1196] Started watching for new ooms in manager
Jul 07 06:31:55.364589 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.364562    2563 manager.go:349] Starting recovery of all containers
Jul 07 06:31:55.372224 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.372010    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 06:31:55.374395 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.374365    2563 manager.go:354] Recovery completed
Jul 07 06:31:55.375620 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:55.375534    2563 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-92.ec2.internal\" not found" node="ip-10-0-133-92.ec2.internal"
Jul 07 06:31:55.376653 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:55.376622    2563 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 07 06:31:55.380119 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.380104    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 06:31:55.382001 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.381985    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 06:31:55.382092 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.382020    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 06:31:55.382092 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.382033    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-92.ec2.internal" event="NodeHasSufficientPID"
Jul 07 06:31:55.382580 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.382566    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 07 06:31:55.382580 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.382578    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 07 06:31:55.382676 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.382600    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 06:31:55.382720 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.382580    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 06:31:55.383444 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.383429    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 06:31:55.383506 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.383460    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 06:31:55.383506 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.383484    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-92.ec2.internal" event="NodeHasSufficientPID"
Jul 07 06:31:55.383857 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.383816    2563 policy_none.go:49] "None policy: Start"
Jul 07 06:31:55.383857 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.383859    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 07 06:31:55.383989 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.383878    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 07 06:31:55.385709 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.385687    2563 policy_none.go:47] "Start"
Jul 07 06:31:55.425778 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.425763    2563 manager.go:337] "Starting Device Plugin manager"
Jul 07 06:31:55.425877 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:55.425802    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 07 06:31:55.425877 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.425812    2563 server.go:87] "Starting device plugin registration server"
Jul 07 06:31:55.426119 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.426102    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 07 06:31:55.426242 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.426119    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 07 06:31:55.426242 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.426210    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 07 06:31:55.426345 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.426330    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 07 06:31:55.426409 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.426345    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 07 06:31:55.426919 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:55.426898    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 07 06:31:55.426996 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:55.426958    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-92.ec2.internal\" not found"
Jul 07 06:31:55.501958 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.501917    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 07 06:31:55.503307 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.503285    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 07 06:31:55.503365 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.503343    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 07 06:31:55.503405 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.503376    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 07 06:31:55.503452 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:55.503419    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 07 06:31:55.505981 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.505954    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 06:31:55.526443 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.526361    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 06:31:55.527384 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.527365    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 06:31:55.527449 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.527401    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 06:31:55.527449 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.527411    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-92.ec2.internal" event="NodeHasSufficientPID"
Jul 07 06:31:55.527449 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.527436    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-92.ec2.internal"
Jul 07 06:31:55.536079 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.536049    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-92.ec2.internal"
Jul 07 06:31:55.536132 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:55.536084    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-92.ec2.internal\": node \"ip-10-0-133-92.ec2.internal\" not found"
Jul 07 06:31:55.549619 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:55.549593    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-92.ec2.internal\" not found"
Jul 07 06:31:55.604314 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.604268    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-92.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-92.ec2.internal"]
Jul 07 06:31:55.604387 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.604372    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 06:31:55.610200 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.610178    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 06:31:55.610289 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.610211    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 06:31:55.610289 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.610221    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-92.ec2.internal" event="NodeHasSufficientPID"
Jul 07 06:31:55.611814 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.611802    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 06:31:55.611925 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.611909    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-92.ec2.internal"
Jul 07 06:31:55.611968 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.611940    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 06:31:55.612664 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.612648    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 06:31:55.612734 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.612680    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 06:31:55.612734 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.612689    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-92.ec2.internal" event="NodeHasSufficientPID"
Jul 07 06:31:55.612795 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.612655    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 06:31:55.612795 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.612754    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 06:31:55.612795 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.612768    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-92.ec2.internal" event="NodeHasSufficientPID"
Jul 07 06:31:55.614479 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.614461    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-92.ec2.internal"
Jul 07 06:31:55.614571 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.614491    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 06:31:55.615255 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.615238    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 06:31:55.615353 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.615270    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 06:31:55.615353 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.615285    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-92.ec2.internal" event="NodeHasSufficientPID"
Jul 07 06:31:55.642760 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:55.642733    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-92.ec2.internal\" not found" node="ip-10-0-133-92.ec2.internal"
Jul 07 06:31:55.647201 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:55.647183    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-92.ec2.internal\" not found" node="ip-10-0-133-92.ec2.internal"
Jul 07 06:31:55.650223 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:55.650208    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-92.ec2.internal\" not found"
Jul 07 06:31:55.750612 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:55.750582    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-92.ec2.internal\" not found"
Jul 07 06:31:55.764048 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.764007    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9ed9ea7211a1e46135e9e4a3c3edc4de-config\") pod \"kube-apiserver-proxy-ip-10-0-133-92.ec2.internal\" (UID: \"9ed9ea7211a1e46135e9e4a3c3edc4de\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-92.ec2.internal"
Jul 07 06:31:55.764184 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.764058    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/10974f874b8e23ee02e9135453ab4de0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-92.ec2.internal\" (UID: \"10974f874b8e23ee02e9135453ab4de0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-92.ec2.internal"
Jul 07 06:31:55.764184 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.764085    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10974f874b8e23ee02e9135453ab4de0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-92.ec2.internal\" (UID: \"10974f874b8e23ee02e9135453ab4de0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-92.ec2.internal"
Jul 07 06:31:55.851404 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:55.851323    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-92.ec2.internal\" not found"
Jul 07 06:31:55.864693 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.864657    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9ed9ea7211a1e46135e9e4a3c3edc4de-config\") pod \"kube-apiserver-proxy-ip-10-0-133-92.ec2.internal\" (UID: \"9ed9ea7211a1e46135e9e4a3c3edc4de\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-92.ec2.internal"
Jul 07 06:31:55.864797 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.864699    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/10974f874b8e23ee02e9135453ab4de0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-92.ec2.internal\" (UID: \"10974f874b8e23ee02e9135453ab4de0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-92.ec2.internal"
Jul 07 06:31:55.864797 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.864719    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10974f874b8e23ee02e9135453ab4de0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-92.ec2.internal\" (UID: \"10974f874b8e23ee02e9135453ab4de0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-92.ec2.internal"
Jul 07 06:31:55.864797 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.864775    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10974f874b8e23ee02e9135453ab4de0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-92.ec2.internal\" (UID: \"10974f874b8e23ee02e9135453ab4de0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-92.ec2.internal"
Jul 07 06:31:55.864932 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.864789    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/10974f874b8e23ee02e9135453ab4de0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-92.ec2.internal\" (UID: \"10974f874b8e23ee02e9135453ab4de0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-92.ec2.internal"
Jul 07 06:31:55.864932 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.864846    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9ed9ea7211a1e46135e9e4a3c3edc4de-config\") pod \"kube-apiserver-proxy-ip-10-0-133-92.ec2.internal\" (UID: \"9ed9ea7211a1e46135e9e4a3c3edc4de\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-92.ec2.internal"
Jul 07 06:31:55.945888 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.945847    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-92.ec2.internal"
Jul 07 06:31:55.950524 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:55.950505    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-92.ec2.internal"
Jul 07 06:31:55.951612 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:55.951593    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-92.ec2.internal\" not found"
Jul 07 06:31:56.051934 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:56.051889    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-92.ec2.internal\" not found"
Jul 07 06:31:57.814304 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:57.814220    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-92.ec2.internal\" not found"
Jul 07 06:31:57.914881 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:57.914838    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-92.ec2.internal\" not found"
Jul 07 06:31:57.957399 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:57.957367    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 07 06:31:57.958050 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:57.957561    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 06:31:57.958050 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:57.957568    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 06:31:58.015881 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:58.015843    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-92.ec2.internal\" not found"
Jul 07 06:31:58.023871 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:58.023831    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 06:26:57 +0000 UTC" deadline="2028-01-22 19:57:31.493844921 +0000 UTC"
Jul 07 06:31:58.023871 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:58.023870    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13549h25m33.469981993s"
Jul 07 06:31:58.109004 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:58.108923    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 06:31:58.124486 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:58.124458    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-92.ec2.internal"
Jul 07 06:31:58.135223 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:58.135195    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 06:31:58.137669 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:58.137631    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-92.ec2.internal"
Jul 07 06:31:58.146050 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:58.146027    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 06:31:58.161443 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:58.161411    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 06:31:58.299868 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:58.299827    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed9ea7211a1e46135e9e4a3c3edc4de.slice/crio-bcf754f675b133d2a480c654dae7fcea3857d54b690729e9a5d570a553ccd227 WatchSource:0}: Error finding container bcf754f675b133d2a480c654dae7fcea3857d54b690729e9a5d570a553ccd227: Status 404 returned error can't find the container with id bcf754f675b133d2a480c654dae7fcea3857d54b690729e9a5d570a553ccd227
Jul 07 06:31:58.300162 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:31:58.300141    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10974f874b8e23ee02e9135453ab4de0.slice/crio-9e25d4b58b664e783d8ea88a88a611bc8c878c0cdd591c98996f85971f62bbb0 WatchSource:0}: Error finding container 9e25d4b58b664e783d8ea88a88a611bc8c878c0cdd591c98996f85971f62bbb0: Status 404 returned error can't find the container with id 9e25d4b58b664e783d8ea88a88a611bc8c878c0cdd591c98996f85971f62bbb0
Jul 07 06:31:58.306767 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:58.306745    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 06:31:59.004033 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.003972    2563 apiserver.go:52] "Watching apiserver"
Jul 07 06:31:59.012449 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.012420    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 07 06:31:59.013783 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.013167    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-xh2bl","kube-system/kube-apiserver-proxy-ip-10-0-133-92.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6","openshift-dns/node-resolver-mbqms","openshift-multus/multus-additional-cni-plugins-b6cs5","openshift-multus/network-metrics-daemon-mx8qb","kube-system/global-pull-secret-syncer-57vh8","openshift-cluster-node-tuning-operator/tuned-sc8n2","openshift-image-registry/node-ca-jg66b","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-92.ec2.internal","openshift-multus/multus-54wzm","openshift-network-diagnostics/network-check-target-jlprw","openshift-network-operator/iptables-alerter-kh9w8","openshift-ovn-kubernetes/ovnkube-node-wnljf"]
Jul 07 06:31:59.015056 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.015033    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:31:59.015176 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:59.015114    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57vh8" podUID="ee9fd62f-81cf-4544-8454-9fa9f53c0768"
Jul 07 06:31:59.016167 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.016143    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kh9w8"
Jul 07 06:31:59.018071 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.017626    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6"
Jul 07 06:31:59.018463 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.018268    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 07 06:31:59.018463 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.018281    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 07 06:31:59.018463 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.018383    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 07 06:31:59.018676 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.018568    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pr94r\""
Jul 07 06:31:59.019421 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.019275    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 07 06:31:59.019421 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.019296    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 07 06:31:59.019878 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.019555    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mbqms"
Jul 07 06:31:59.021016 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.020902    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vxfsc\""
Jul 07 06:31:59.021016 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.020962    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 07 06:31:59.021176 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.021090    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b6cs5"
Jul 07 06:31:59.021867 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.021394    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4tg7h\""
Jul 07 06:31:59.021867 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.021444    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 07 06:31:59.021867 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.021709    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 07 06:31:59.022952 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.022931    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 07 06:31:59.023039 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.022978    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 07 06:31:59.023337 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.023211    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9zldw\""
Jul 07 06:31:59.023337 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.023285    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 07 06:31:59.023506 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.023364    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 07 06:31:59.023617 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.023598    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 07 06:31:59.023710 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.023616    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:31:59.023768 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:59.023748    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx8qb" podUID="7c1e103c-f156-4b8a-8426-f18974595c51"
Jul 07 06:31:59.023869 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.023853    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 07 06:31:59.027180 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.024904    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jg66b"
Jul 07 06:31:59.027180 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.026122    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xh2bl"
Jul 07 06:31:59.028509 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.028485    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 07 06:31:59.028590 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.028549    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 07 06:31:59.028883 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.028867    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 07 06:31:59.029106 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.029086    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 07 06:31:59.029171 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.029150    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 07 06:31:59.029316 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.029302    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jj4sd\""
Jul 07 06:31:59.030464 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.029517    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-k6plg\""
Jul 07 06:31:59.030464 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.029963    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.034119 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.032010    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 07 06:31:59.034119 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.032269    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 07 06:31:59.034119 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.032344    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.034119 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.032810    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 07 06:31:59.034119 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.032830    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 07 06:31:59.034119 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.033057    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 07 06:31:59.034119 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.033270    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 07 06:31:59.034119 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.033390    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m97kb\""
Jul 07 06:31:59.035654 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.035618    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.035747 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.035661    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce422fc1-946c-45e5-b5bf-2203adc5f7f9-etc-selinux\") pod \"aws-ebs-csi-driver-node-lqlp6\" (UID: \"ce422fc1-946c-45e5-b5bf-2203adc5f7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6"
Jul 07 06:31:59.035747 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.035691    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4aaa39b2-a25e-4a3c-b3b8-b65d0b6d140b-hosts-file\") pod \"node-resolver-mbqms\" (UID: \"4aaa39b2-a25e-4a3c-b3b8-b65d0b6d140b\") " pod="openshift-dns/node-resolver-mbqms"
Jul 07 06:31:59.035747 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.035717    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c-system-cni-dir\") pod \"multus-additional-cni-plugins-b6cs5\" (UID: \"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c\") " pod="openshift-multus/multus-additional-cni-plugins-b6cs5"
Jul 07 06:31:59.035747 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.035742    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b6cs5\" (UID: \"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c\") " pod="openshift-multus/multus-additional-cni-plugins-b6cs5"
Jul 07 06:31:59.037004 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.035765    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ee9fd62f-81cf-4544-8454-9fa9f53c0768-dbus\") pod \"global-pull-secret-syncer-57vh8\" (UID: \"ee9fd62f-81cf-4544-8454-9fa9f53c0768\") " pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:31:59.037004 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.035756    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 07 06:31:59.037004 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.035842    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce422fc1-946c-45e5-b5bf-2203adc5f7f9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lqlp6\" (UID: \"ce422fc1-946c-45e5-b5bf-2203adc5f7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6"
Jul 07 06:31:59.037004 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.035869    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce422fc1-946c-45e5-b5bf-2203adc5f7f9-socket-dir\") pod \"aws-ebs-csi-driver-node-lqlp6\" (UID: \"ce422fc1-946c-45e5-b5bf-2203adc5f7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6"
Jul 07 06:31:59.037004 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.036023    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xz9j9\""
Jul 07 06:31:59.037004 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.036078    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 07 06:31:59.037004 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.036077    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c-os-release\") pod \"multus-additional-cni-plugins-b6cs5\" (UID: \"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c\") " pod="openshift-multus/multus-additional-cni-plugins-b6cs5"
Jul 07 06:31:59.037004 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.036169    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b6cs5\" (UID: \"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c\") " pod="openshift-multus/multus-additional-cni-plugins-b6cs5"
Jul 07 06:31:59.037004 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.036197    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srqj6\" (UniqueName: \"kubernetes.io/projected/0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c-kube-api-access-srqj6\") pod \"multus-additional-cni-plugins-b6cs5\" (UID: \"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c\") " pod="openshift-multus/multus-additional-cni-plugins-b6cs5"
Jul 07 06:31:59.037004 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.036223    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgzg2\" (UniqueName: \"kubernetes.io/projected/7c1e103c-f156-4b8a-8426-f18974595c51-kube-api-access-sgzg2\") pod \"network-metrics-daemon-mx8qb\" (UID: \"7c1e103c-f156-4b8a-8426-f18974595c51\") " pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:31:59.037004 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.036247    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/327fd7db-21be-45ef-9efd-6c3f46c1e663-serviceca\") pod \"node-ca-jg66b\" (UID: \"327fd7db-21be-45ef-9efd-6c3f46c1e663\") " pod="openshift-image-registry/node-ca-jg66b"
Jul 07 06:31:59.037004 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.036267    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aaa39b2-a25e-4a3c-b3b8-b65d0b6d140b-tmp-dir\") pod \"node-resolver-mbqms\" (UID: \"4aaa39b2-a25e-4a3c-b3b8-b65d0b6d140b\") " pod="openshift-dns/node-resolver-mbqms"
Jul 07 06:31:59.037004 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.036289    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c1e103c-f156-4b8a-8426-f18974595c51-metrics-certs\") pod \"network-metrics-daemon-mx8qb\" (UID: \"7c1e103c-f156-4b8a-8426-f18974595c51\") " pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:31:59.037004 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.036390    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ee9fd62f-81cf-4544-8454-9fa9f53c0768-kubelet-config\") pod \"global-pull-secret-syncer-57vh8\" (UID: \"ee9fd62f-81cf-4544-8454-9fa9f53c0768\") " pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:31:59.037004 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.036421    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a9cb87a-eb89-462d-b9dc-83e0f39a712d-host-slash\") pod \"iptables-alerter-kh9w8\" (UID: \"1a9cb87a-eb89-462d-b9dc-83e0f39a712d\") " pod="openshift-network-operator/iptables-alerter-kh9w8"
Jul 07 06:31:59.037004 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.036447    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/327fd7db-21be-45ef-9efd-6c3f46c1e663-host\") pod \"node-ca-jg66b\" (UID: \"327fd7db-21be-45ef-9efd-6c3f46c1e663\") " pod="openshift-image-registry/node-ca-jg66b"
Jul 07 06:31:59.037004 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.036473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmmqr\" (UniqueName: \"kubernetes.io/projected/327fd7db-21be-45ef-9efd-6c3f46c1e663-kube-api-access-gmmqr\") pod \"node-ca-jg66b\" (UID: \"327fd7db-21be-45ef-9efd-6c3f46c1e663\") " pod="openshift-image-registry/node-ca-jg66b"
Jul 07 06:31:59.037004 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.036498    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce422fc1-946c-45e5-b5bf-2203adc5f7f9-registration-dir\") pod \"aws-ebs-csi-driver-node-lqlp6\" (UID: \"ce422fc1-946c-45e5-b5bf-2203adc5f7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6"
Jul 07 06:31:59.037963 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.036508    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:31:59.037963 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.036522    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce422fc1-946c-45e5-b5bf-2203adc5f7f9-device-dir\") pod \"aws-ebs-csi-driver-node-lqlp6\" (UID: \"ce422fc1-946c-45e5-b5bf-2203adc5f7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6"
Jul 07 06:31:59.037963 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.036547    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce422fc1-946c-45e5-b5bf-2203adc5f7f9-sys-fs\") pod \"aws-ebs-csi-driver-node-lqlp6\" (UID: \"ce422fc1-946c-45e5-b5bf-2203adc5f7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6"
Jul 07 06:31:59.037963 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:59.036571    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlprw" podUID="ffb473f1-6af6-40d5-b15b-23939d3eff20"
Jul 07 06:31:59.037963 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.036585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee9fd62f-81cf-4544-8454-9fa9f53c0768-original-pull-secret\") pod \"global-pull-secret-syncer-57vh8\" (UID: \"ee9fd62f-81cf-4544-8454-9fa9f53c0768\") " pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:31:59.037963 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.036651    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqchk\" (UniqueName: \"kubernetes.io/projected/1a9cb87a-eb89-462d-b9dc-83e0f39a712d-kube-api-access-dqchk\") pod \"iptables-alerter-kh9w8\" (UID: \"1a9cb87a-eb89-462d-b9dc-83e0f39a712d\") " pod="openshift-network-operator/iptables-alerter-kh9w8"
Jul 07 06:31:59.037963 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.036692    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgv6v\" (UniqueName: \"kubernetes.io/projected/ce422fc1-946c-45e5-b5bf-2203adc5f7f9-kube-api-access-mgv6v\") pod \"aws-ebs-csi-driver-node-lqlp6\" (UID: \"ce422fc1-946c-45e5-b5bf-2203adc5f7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6"
Jul 07 06:31:59.037963 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.036721    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c-cnibin\") pod \"multus-additional-cni-plugins-b6cs5\" (UID: \"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c\") " pod="openshift-multus/multus-additional-cni-plugins-b6cs5"
Jul 07 06:31:59.037963 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.036748    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1a9cb87a-eb89-462d-b9dc-83e0f39a712d-iptables-alerter-script\") pod \"iptables-alerter-kh9w8\" (UID: \"1a9cb87a-eb89-462d-b9dc-83e0f39a712d\") " pod="openshift-network-operator/iptables-alerter-kh9w8"
Jul 07 06:31:59.037963 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.036773    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7f8g\" (UniqueName: \"kubernetes.io/projected/4aaa39b2-a25e-4a3c-b3b8-b65d0b6d140b-kube-api-access-z7f8g\") pod \"node-resolver-mbqms\" (UID: \"4aaa39b2-a25e-4a3c-b3b8-b65d0b6d140b\") " pod="openshift-dns/node-resolver-mbqms"
Jul 07 06:31:59.037963 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.036798    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c-cni-binary-copy\") pod \"multus-additional-cni-plugins-b6cs5\" (UID: \"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c\") " pod="openshift-multus/multus-additional-cni-plugins-b6cs5"
Jul 07 06:31:59.037963 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.036824    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b6cs5\" (UID: \"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c\") " pod="openshift-multus/multus-additional-cni-plugins-b6cs5"
Jul 07 06:31:59.055581 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.050736    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 06:31:59.055581 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.051010    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 07 06:31:59.055581 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.051203    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6rq4j\""
Jul 07 06:31:59.059063 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.058995    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-d5g5h"
Jul 07 06:31:59.066134 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.066063    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-d5g5h"
Jul 07 06:31:59.125689 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.125657    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 07 06:31:59.137339 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.137291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmmqr\" (UniqueName: \"kubernetes.io/projected/327fd7db-21be-45ef-9efd-6c3f46c1e663-kube-api-access-gmmqr\") pod \"node-ca-jg66b\" (UID: \"327fd7db-21be-45ef-9efd-6c3f46c1e663\") " pod="openshift-image-registry/node-ca-jg66b"
Jul 07 06:31:59.137522 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.137354    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-var-lib-openvswitch\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.137522 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.137380    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-node-log\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.137522 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.137400    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-etc-sysctl-conf\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.137522 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.137428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce422fc1-946c-45e5-b5bf-2203adc5f7f9-registration-dir\") pod \"aws-ebs-csi-driver-node-lqlp6\" (UID: \"ce422fc1-946c-45e5-b5bf-2203adc5f7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6"
Jul 07 06:31:59.137522 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.137454    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce422fc1-946c-45e5-b5bf-2203adc5f7f9-sys-fs\") pod \"aws-ebs-csi-driver-node-lqlp6\" (UID: \"ce422fc1-946c-45e5-b5bf-2203adc5f7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6"
Jul 07 06:31:59.137522 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.137480    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee9fd62f-81cf-4544-8454-9fa9f53c0768-original-pull-secret\") pod \"global-pull-secret-syncer-57vh8\" (UID: \"ee9fd62f-81cf-4544-8454-9fa9f53c0768\") " pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:31:59.137522 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.137502    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-var-lib-kubelet\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.137522 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.137523    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a649f06f-00a6-4ae6-8e68-ce87925329da-etc-tuned\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.138075 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.137613    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce422fc1-946c-45e5-b5bf-2203adc5f7f9-sys-fs\") pod \"aws-ebs-csi-driver-node-lqlp6\" (UID: \"ce422fc1-946c-45e5-b5bf-2203adc5f7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6"
Jul 07 06:31:59.138075 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.137613    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce422fc1-946c-45e5-b5bf-2203adc5f7f9-registration-dir\") pod \"aws-ebs-csi-driver-node-lqlp6\" (UID: \"ce422fc1-946c-45e5-b5bf-2203adc5f7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6"
Jul 07 06:31:59.138075 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.137667    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgv6v\" (UniqueName: \"kubernetes.io/projected/ce422fc1-946c-45e5-b5bf-2203adc5f7f9-kube-api-access-mgv6v\") pod \"aws-ebs-csi-driver-node-lqlp6\" (UID: \"ce422fc1-946c-45e5-b5bf-2203adc5f7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6"
Jul 07 06:31:59.138075 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.137706    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-host-slash\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.138075 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.137731    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-env-overrides\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.138075 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:59.137748    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 06:31:59.138075 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.137775    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-run\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.138075 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:59.137829    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee9fd62f-81cf-4544-8454-9fa9f53c0768-original-pull-secret podName:ee9fd62f-81cf-4544-8454-9fa9f53c0768 nodeName:}" failed. No retries permitted until 2026-07-07 06:31:59.637793115 +0000 UTC m=+3.001989642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ee9fd62f-81cf-4544-8454-9fa9f53c0768-original-pull-secret") pod "global-pull-secret-syncer-57vh8" (UID: "ee9fd62f-81cf-4544-8454-9fa9f53c0768") : object "kube-system"/"original-pull-secret" not registered
Jul 07 06:31:59.138075 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.137851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-host-run-multus-certs\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.138075 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.137885    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c-cnibin\") pod \"multus-additional-cni-plugins-b6cs5\" (UID: \"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c\") " pod="openshift-multus/multus-additional-cni-plugins-b6cs5"
Jul 07 06:31:59.138075 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.137908    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1a9cb87a-eb89-462d-b9dc-83e0f39a712d-iptables-alerter-script\") pod \"iptables-alerter-kh9w8\" (UID: \"1a9cb87a-eb89-462d-b9dc-83e0f39a712d\") " pod="openshift-network-operator/iptables-alerter-kh9w8"
Jul 07 06:31:59.138075 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.137936    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.138075 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.137984    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-etc-kubernetes\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.138075 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138020    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-etc-sysctl-d\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.138075 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138044    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c-cnibin\") pod \"multus-additional-cni-plugins-b6cs5\" (UID: \"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c\") " pod="openshift-multus/multus-additional-cni-plugins-b6cs5"
Jul 07 06:31:59.138075 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138047    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2060cba3-4269-4ecd-a03c-7a51fad57d0c-cni-binary-copy\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.138908 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138086    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-etc-kubernetes\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.138908 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138110    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7f8g\" (UniqueName: \"kubernetes.io/projected/4aaa39b2-a25e-4a3c-b3b8-b65d0b6d140b-kube-api-access-z7f8g\") pod \"node-resolver-mbqms\" (UID: \"4aaa39b2-a25e-4a3c-b3b8-b65d0b6d140b\") " pod="openshift-dns/node-resolver-mbqms"
Jul 07 06:31:59.138908 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138147    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c-cni-binary-copy\") pod \"multus-additional-cni-plugins-b6cs5\" (UID: \"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c\") " pod="openshift-multus/multus-additional-cni-plugins-b6cs5"
Jul 07 06:31:59.138908 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138162    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-host-run-netns\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.138908 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138180    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-log-socket\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.138908 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138203    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-host-run-ovn-kubernetes\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.138908 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138223    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-host-cni-bin\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.138908 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138237    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-lib-modules\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.138908 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138265    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-multus-socket-dir-parent\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.138908 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138289    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c-system-cni-dir\") pod \"multus-additional-cni-plugins-b6cs5\" (UID: \"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c\") " pod="openshift-multus/multus-additional-cni-plugins-b6cs5"
Jul 07 06:31:59.138908 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138317    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ee9fd62f-81cf-4544-8454-9fa9f53c0768-dbus\") pod \"global-pull-secret-syncer-57vh8\" (UID: \"ee9fd62f-81cf-4544-8454-9fa9f53c0768\") " pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:31:59.138908 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-etc-sysconfig\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.138908 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138356    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-host\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.138908 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-multus-cni-dir\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.138908 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138395    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce422fc1-946c-45e5-b5bf-2203adc5f7f9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lqlp6\" (UID: \"ce422fc1-946c-45e5-b5bf-2203adc5f7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6"
Jul 07 06:31:59.138908 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138411    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce422fc1-946c-45e5-b5bf-2203adc5f7f9-socket-dir\") pod \"aws-ebs-csi-driver-node-lqlp6\" (UID: \"ce422fc1-946c-45e5-b5bf-2203adc5f7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6"
Jul 07 06:31:59.138908 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138426    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b6cs5\" (UID: \"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c\") " pod="openshift-multus/multus-additional-cni-plugins-b6cs5"
Jul 07 06:31:59.139724 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srqj6\" (UniqueName: \"kubernetes.io/projected/0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c-kube-api-access-srqj6\") pod \"multus-additional-cni-plugins-b6cs5\" (UID: \"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c\") " pod="openshift-multus/multus-additional-cni-plugins-b6cs5"
Jul 07 06:31:59.139724 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138498    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-ovnkube-config\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.139724 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138554    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1a9cb87a-eb89-462d-b9dc-83e0f39a712d-iptables-alerter-script\") pod \"iptables-alerter-kh9w8\" (UID: \"1a9cb87a-eb89-462d-b9dc-83e0f39a712d\") " pod="openshift-network-operator/iptables-alerter-kh9w8"
Jul 07 06:31:59.139724 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138606    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-ovn-node-metrics-cert\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.139724 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-system-cni-dir\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.139724 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138677    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c1e103c-f156-4b8a-8426-f18974595c51-metrics-certs\") pod \"network-metrics-daemon-mx8qb\" (UID: \"7c1e103c-f156-4b8a-8426-f18974595c51\") " pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:31:59.139724 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138706    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ee9fd62f-81cf-4544-8454-9fa9f53c0768-kubelet-config\") pod \"global-pull-secret-syncer-57vh8\" (UID: \"ee9fd62f-81cf-4544-8454-9fa9f53c0768\") " pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:31:59.139724 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138731    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-run-systemd\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.139724 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138770    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c5d1a464-df78-4f33-8ae7-f81183a9fb39-agent-certs\") pod \"konnectivity-agent-xh2bl\" (UID: \"c5d1a464-df78-4f33-8ae7-f81183a9fb39\") " pod="kube-system/konnectivity-agent-xh2bl"
Jul 07 06:31:59.139724 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138796    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-etc-modprobe-d\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.139724 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-sys\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.139724 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138839    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-os-release\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.139724 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138867    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-host-run-k8s-cni-cncf-io\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.139724 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138893    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/327fd7db-21be-45ef-9efd-6c3f46c1e663-host\") pod \"node-ca-jg66b\" (UID: \"327fd7db-21be-45ef-9efd-6c3f46c1e663\") " pod="openshift-image-registry/node-ca-jg66b"
Jul 07 06:31:59.139724 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.138921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-etc-systemd\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.139724 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.139050    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6khjs\" (UniqueName: \"kubernetes.io/projected/a649f06f-00a6-4ae6-8e68-ce87925329da-kube-api-access-6khjs\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.139724 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.139077    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-cnibin\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.140461 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.139103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2060cba3-4269-4ecd-a03c-7a51fad57d0c-multus-daemon-config\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.140461 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.139122    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c-cni-binary-copy\") pod \"multus-additional-cni-plugins-b6cs5\" (UID: \"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c\") " pod="openshift-multus/multus-additional-cni-plugins-b6cs5"
Jul 07 06:31:59.140461 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.139128    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7qpw\" (UniqueName: \"kubernetes.io/projected/2060cba3-4269-4ecd-a03c-7a51fad57d0c-kube-api-access-v7qpw\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.140461 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.139157    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce422fc1-946c-45e5-b5bf-2203adc5f7f9-device-dir\") pod \"aws-ebs-csi-driver-node-lqlp6\" (UID: \"ce422fc1-946c-45e5-b5bf-2203adc5f7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6"
Jul 07 06:31:59.140461 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.139185    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqchk\" (UniqueName: \"kubernetes.io/projected/1a9cb87a-eb89-462d-b9dc-83e0f39a712d-kube-api-access-dqchk\") pod \"iptables-alerter-kh9w8\" (UID: \"1a9cb87a-eb89-462d-b9dc-83e0f39a712d\") " pod="openshift-network-operator/iptables-alerter-kh9w8"
Jul 07 06:31:59.140461 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.139211    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-etc-openvswitch\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.140461 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.139243    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b6cs5\" (UID: \"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c\") " pod="openshift-multus/multus-additional-cni-plugins-b6cs5"
Jul 07 06:31:59.140461 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.139508    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c-system-cni-dir\") pod \"multus-additional-cni-plugins-b6cs5\" (UID: \"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c\") " pod="openshift-multus/multus-additional-cni-plugins-b6cs5"
Jul 07 06:31:59.140461 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.139728    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-run-openvswitch\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.140461 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.139755    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a649f06f-00a6-4ae6-8e68-ce87925329da-tmp\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.140461 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.139776    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce422fc1-946c-45e5-b5bf-2203adc5f7f9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lqlp6\" (UID: \"ce422fc1-946c-45e5-b5bf-2203adc5f7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6"
Jul 07 06:31:59.140461 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.139805    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-multus-conf-dir\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.140461 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.139883    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/327fd7db-21be-45ef-9efd-6c3f46c1e663-host\") pod \"node-ca-jg66b\" (UID: \"327fd7db-21be-45ef-9efd-6c3f46c1e663\") " pod="openshift-image-registry/node-ca-jg66b"
Jul 07 06:31:59.140461 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.139913    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce422fc1-946c-45e5-b5bf-2203adc5f7f9-socket-dir\") pod \"aws-ebs-csi-driver-node-lqlp6\" (UID: \"ce422fc1-946c-45e5-b5bf-2203adc5f7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6"
Jul 07 06:31:59.140461 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.139990    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce422fc1-946c-45e5-b5bf-2203adc5f7f9-device-dir\") pod \"aws-ebs-csi-driver-node-lqlp6\" (UID: \"ce422fc1-946c-45e5-b5bf-2203adc5f7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6"
Jul 07 06:31:59.140461 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:59.140021    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:31:59.140461 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:59.140081    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c1e103c-f156-4b8a-8426-f18974595c51-metrics-certs podName:7c1e103c-f156-4b8a-8426-f18974595c51 nodeName:}" failed. No retries permitted until 2026-07-07 06:31:59.640059715 +0000 UTC m=+3.004256241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c1e103c-f156-4b8a-8426-f18974595c51-metrics-certs") pod "network-metrics-daemon-mx8qb" (UID: "7c1e103c-f156-4b8a-8426-f18974595c51") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:31:59.141224 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.140143    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ee9fd62f-81cf-4544-8454-9fa9f53c0768-kubelet-config\") pod \"global-pull-secret-syncer-57vh8\" (UID: \"ee9fd62f-81cf-4544-8454-9fa9f53c0768\") " pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:31:59.141224 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.139727    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ee9fd62f-81cf-4544-8454-9fa9f53c0768-dbus\") pod \"global-pull-secret-syncer-57vh8\" (UID: \"ee9fd62f-81cf-4544-8454-9fa9f53c0768\") " pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:31:59.142467 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.140313    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-host-kubelet\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.142726 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.142685    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-host-run-netns\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.142828 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.142750    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-host-var-lib-cni-multus\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.142867 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.142823    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-hostroot\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.142952 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.142908    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b6cs5\" (UID: \"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c\") " pod="openshift-multus/multus-additional-cni-plugins-b6cs5"
Jul 07 06:31:59.143101 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.142993    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m48ld\" (UniqueName: \"kubernetes.io/projected/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-kube-api-access-m48ld\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.143101 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.143026    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce422fc1-946c-45e5-b5bf-2203adc5f7f9-etc-selinux\") pod \"aws-ebs-csi-driver-node-lqlp6\" (UID: \"ce422fc1-946c-45e5-b5bf-2203adc5f7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6"
Jul 07 06:31:59.143101 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.143078    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4aaa39b2-a25e-4a3c-b3b8-b65d0b6d140b-hosts-file\") pod \"node-resolver-mbqms\" (UID: \"4aaa39b2-a25e-4a3c-b3b8-b65d0b6d140b\") " pod="openshift-dns/node-resolver-mbqms"
Jul 07 06:31:59.143236 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.143124    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b6cs5\" (UID: \"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c\") " pod="openshift-multus/multus-additional-cni-plugins-b6cs5"
Jul 07 06:31:59.143236 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.143178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-host-cni-netd\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.143236 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.143209    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c5d1a464-df78-4f33-8ae7-f81183a9fb39-konnectivity-ca\") pod \"konnectivity-agent-xh2bl\" (UID: \"c5d1a464-df78-4f33-8ae7-f81183a9fb39\") " pod="kube-system/konnectivity-agent-xh2bl"
Jul 07 06:31:59.143372 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.143276    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-host-var-lib-cni-bin\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.143372 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.143307    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c-os-release\") pod \"multus-additional-cni-plugins-b6cs5\" (UID: \"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c\") " pod="openshift-multus/multus-additional-cni-plugins-b6cs5"
Jul 07 06:31:59.143372 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.143339    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgzg2\" (UniqueName: \"kubernetes.io/projected/7c1e103c-f156-4b8a-8426-f18974595c51-kube-api-access-sgzg2\") pod \"network-metrics-daemon-mx8qb\" (UID: \"7c1e103c-f156-4b8a-8426-f18974595c51\") " pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:31:59.143372 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.143364    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/327fd7db-21be-45ef-9efd-6c3f46c1e663-serviceca\") pod \"node-ca-jg66b\" (UID: \"327fd7db-21be-45ef-9efd-6c3f46c1e663\") " pod="openshift-image-registry/node-ca-jg66b"
Jul 07 06:31:59.143538 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.143394    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a9cb87a-eb89-462d-b9dc-83e0f39a712d-host-slash\") pod \"iptables-alerter-kh9w8\" (UID: \"1a9cb87a-eb89-462d-b9dc-83e0f39a712d\") " pod="openshift-network-operator/iptables-alerter-kh9w8"
Jul 07 06:31:59.143538 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.143422    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-run-ovn\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.143538 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.143467    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-host-var-lib-kubelet\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.143538 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.143508    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47ljz\" (UniqueName: \"kubernetes.io/projected/ffb473f1-6af6-40d5-b15b-23939d3eff20-kube-api-access-47ljz\") pod \"network-check-target-jlprw\" (UID: \"ffb473f1-6af6-40d5-b15b-23939d3eff20\") " pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:31:59.143783 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.143745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aaa39b2-a25e-4a3c-b3b8-b65d0b6d140b-tmp-dir\") pod \"node-resolver-mbqms\" (UID: \"4aaa39b2-a25e-4a3c-b3b8-b65d0b6d140b\") " pod="openshift-dns/node-resolver-mbqms"
Jul 07 06:31:59.143783 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.143778    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-systemd-units\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.143888 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.143821    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-ovnkube-script-lib\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.144101 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.144078    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce422fc1-946c-45e5-b5bf-2203adc5f7f9-etc-selinux\") pod \"aws-ebs-csi-driver-node-lqlp6\" (UID: \"ce422fc1-946c-45e5-b5bf-2203adc5f7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6"
Jul 07 06:31:59.144188 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.144160    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 07 06:31:59.144497 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.144471    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/327fd7db-21be-45ef-9efd-6c3f46c1e663-serviceca\") pod \"node-ca-jg66b\" (UID: \"327fd7db-21be-45ef-9efd-6c3f46c1e663\") " pod="openshift-image-registry/node-ca-jg66b"
Jul 07 06:31:59.144573 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.144537    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a9cb87a-eb89-462d-b9dc-83e0f39a712d-host-slash\") pod \"iptables-alerter-kh9w8\" (UID: \"1a9cb87a-eb89-462d-b9dc-83e0f39a712d\") " pod="openshift-network-operator/iptables-alerter-kh9w8"
Jul 07 06:31:59.144655 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.144617    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b6cs5\" (UID: \"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c\") " pod="openshift-multus/multus-additional-cni-plugins-b6cs5"
Jul 07 06:31:59.144762 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.144612    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c-os-release\") pod \"multus-additional-cni-plugins-b6cs5\" (UID: \"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c\") " pod="openshift-multus/multus-additional-cni-plugins-b6cs5"
Jul 07 06:31:59.144902 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.144884    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aaa39b2-a25e-4a3c-b3b8-b65d0b6d140b-tmp-dir\") pod \"node-resolver-mbqms\" (UID: \"4aaa39b2-a25e-4a3c-b3b8-b65d0b6d140b\") " pod="openshift-dns/node-resolver-mbqms"
Jul 07 06:31:59.145057 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.145034    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4aaa39b2-a25e-4a3c-b3b8-b65d0b6d140b-hosts-file\") pod \"node-resolver-mbqms\" (UID: \"4aaa39b2-a25e-4a3c-b3b8-b65d0b6d140b\") " pod="openshift-dns/node-resolver-mbqms"
Jul 07 06:31:59.145165 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.145152    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b6cs5\" (UID: \"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c\") " pod="openshift-multus/multus-additional-cni-plugins-b6cs5"
Jul 07 06:31:59.150136 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.150109    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgv6v\" (UniqueName: \"kubernetes.io/projected/ce422fc1-946c-45e5-b5bf-2203adc5f7f9-kube-api-access-mgv6v\") pod \"aws-ebs-csi-driver-node-lqlp6\" (UID: \"ce422fc1-946c-45e5-b5bf-2203adc5f7f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6"
Jul 07 06:31:59.150404 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.150241    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmmqr\" (UniqueName: \"kubernetes.io/projected/327fd7db-21be-45ef-9efd-6c3f46c1e663-kube-api-access-gmmqr\") pod \"node-ca-jg66b\" (UID: \"327fd7db-21be-45ef-9efd-6c3f46c1e663\") " pod="openshift-image-registry/node-ca-jg66b"
Jul 07 06:31:59.150404 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.150281    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7f8g\" (UniqueName: \"kubernetes.io/projected/4aaa39b2-a25e-4a3c-b3b8-b65d0b6d140b-kube-api-access-z7f8g\") pod \"node-resolver-mbqms\" (UID: \"4aaa39b2-a25e-4a3c-b3b8-b65d0b6d140b\") " pod="openshift-dns/node-resolver-mbqms"
Jul 07 06:31:59.151007 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.150985    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srqj6\" (UniqueName: \"kubernetes.io/projected/0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c-kube-api-access-srqj6\") pod \"multus-additional-cni-plugins-b6cs5\" (UID: \"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c\") " pod="openshift-multus/multus-additional-cni-plugins-b6cs5"
Jul 07 06:31:59.152318 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.152226    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqchk\" (UniqueName: \"kubernetes.io/projected/1a9cb87a-eb89-462d-b9dc-83e0f39a712d-kube-api-access-dqchk\") pod \"iptables-alerter-kh9w8\" (UID: \"1a9cb87a-eb89-462d-b9dc-83e0f39a712d\") " pod="openshift-network-operator/iptables-alerter-kh9w8"
Jul 07 06:31:59.153344 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.153318    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgzg2\" (UniqueName: \"kubernetes.io/projected/7c1e103c-f156-4b8a-8426-f18974595c51-kube-api-access-sgzg2\") pod \"network-metrics-daemon-mx8qb\" (UID: \"7c1e103c-f156-4b8a-8426-f18974595c51\") " pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:31:59.172147 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.171794    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-92.ec2.internal" event={"ID":"10974f874b8e23ee02e9135453ab4de0","Type":"ContainerStarted","Data":"9e25d4b58b664e783d8ea88a88a611bc8c878c0cdd591c98996f85971f62bbb0"}
Jul 07 06:31:59.174806 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.174776    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-92.ec2.internal" event={"ID":"9ed9ea7211a1e46135e9e4a3c3edc4de","Type":"ContainerStarted","Data":"bcf754f675b133d2a480c654dae7fcea3857d54b690729e9a5d570a553ccd227"}
Jul 07 06:31:59.244818 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.244782    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-systemd-units\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.245001 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.244826    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-ovnkube-script-lib\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.245001 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.244853    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-var-lib-openvswitch\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.245001 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.244877    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-node-log\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.245001 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.244903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-etc-sysctl-conf\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.245001 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.244931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-var-lib-kubelet\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.245001 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.244945    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a649f06f-00a6-4ae6-8e68-ce87925329da-etc-tuned\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.245001 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.244969    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-host-slash\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.245001 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.244987    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-env-overrides\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.245001 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-run\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.245467 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245026    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-host-run-multus-certs\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.245467 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245051    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.245467 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245083    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-etc-kubernetes\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.245467 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245098    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-etc-sysctl-d\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.245467 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245116    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2060cba3-4269-4ecd-a03c-7a51fad57d0c-cni-binary-copy\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.245467 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245140    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-etc-kubernetes\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.245467 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245156    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-host-run-netns\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.245467 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245169    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-var-lib-kubelet\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.245467 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245178    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-log-socket\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.245467 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245220    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-log-socket\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.245467 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245232    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-host-run-ovn-kubernetes\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.245467 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245275    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-host-cni-bin\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.245467 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245300    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-lib-modules\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.245467 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245326    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-multus-socket-dir-parent\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.245467 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245352    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-etc-sysconfig\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.245467 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245376    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-host\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.245467 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-multus-cni-dir\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.246248 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245434    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-ovnkube-config\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.246248 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245460    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-ovn-node-metrics-cert\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.246248 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245484    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-system-cni-dir\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.246248 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245525    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-run-systemd\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.246248 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245552    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c5d1a464-df78-4f33-8ae7-f81183a9fb39-agent-certs\") pod \"konnectivity-agent-xh2bl\" (UID: \"c5d1a464-df78-4f33-8ae7-f81183a9fb39\") " pod="kube-system/konnectivity-agent-xh2bl"
Jul 07 06:31:59.246248 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245579    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-etc-modprobe-d\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.246248 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245601    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.246248 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245652    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-multus-socket-dir-parent\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.246248 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245668    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-etc-kubernetes\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.246248 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245678    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-host-slash\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.246248 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245739    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-system-cni-dir\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.246248 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245739    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-etc-sysconfig\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.246248 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245739    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-etc-sysctl-d\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.246248 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245788    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-etc-modprobe-d\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.246248 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245829    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-run-systemd\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.246248 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245857    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-etc-kubernetes\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.246248 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245887    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-run\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.246248 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.245894    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-multus-cni-dir\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.246836 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246151    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-etc-sysctl-conf\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.246836 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246195    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-systemd-units\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.246836 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246245    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-env-overrides\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.246836 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246251    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-ovnkube-config\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.246836 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246251    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2060cba3-4269-4ecd-a03c-7a51fad57d0c-cni-binary-copy\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.246836 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246291    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-host-cni-bin\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.246836 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246298    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-host-run-multus-certs\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.246836 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246344    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-host\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.246836 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246371    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-node-log\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.246836 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246397    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-host-run-ovn-kubernetes\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.246836 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246406    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-var-lib-openvswitch\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.246836 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-sys\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.246836 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246438    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-host-run-netns\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.246836 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246413    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-lib-modules\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.246836 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246472    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-os-release\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.246836 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246502    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-host-run-k8s-cni-cncf-io\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.246836 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-etc-systemd\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.246836 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246594    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-os-release\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.248027 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246627    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-sys\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.248027 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6khjs\" (UniqueName: \"kubernetes.io/projected/a649f06f-00a6-4ae6-8e68-ce87925329da-kube-api-access-6khjs\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.248027 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246677    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-cnibin\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.248027 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246703    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2060cba3-4269-4ecd-a03c-7a51fad57d0c-multus-daemon-config\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.248027 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246707    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-host-run-k8s-cni-cncf-io\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.248027 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246735    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-ovnkube-script-lib\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.248027 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7qpw\" (UniqueName: \"kubernetes.io/projected/2060cba3-4269-4ecd-a03c-7a51fad57d0c-kube-api-access-v7qpw\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.248027 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246788    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-etc-openvswitch\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.248027 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246795    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a649f06f-00a6-4ae6-8e68-ce87925329da-etc-systemd\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.248027 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246833    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-run-openvswitch\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.248027 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246848    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-cnibin\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.248027 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a649f06f-00a6-4ae6-8e68-ce87925329da-tmp\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.248027 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246892    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-etc-openvswitch\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.248027 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-multus-conf-dir\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.248027 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246941    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-host-kubelet\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.248027 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.246967    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-host-run-netns\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.248027 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.247002    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-host-var-lib-cni-multus\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.248027 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.247026    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-hostroot\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.248931 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.247051    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m48ld\" (UniqueName: \"kubernetes.io/projected/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-kube-api-access-m48ld\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.248931 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.247091    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-host-run-netns\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.248931 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.247124    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-multus-conf-dir\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.248931 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.247164    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-host-kubelet\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.248931 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.247201    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-host-var-lib-cni-multus\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.248931 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.247254    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-run-openvswitch\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.248931 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.247299    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-hostroot\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.248931 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.247550    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-host-cni-netd\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.248931 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.247586    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c5d1a464-df78-4f33-8ae7-f81183a9fb39-konnectivity-ca\") pod \"konnectivity-agent-xh2bl\" (UID: \"c5d1a464-df78-4f33-8ae7-f81183a9fb39\") " pod="kube-system/konnectivity-agent-xh2bl"
Jul 07 06:31:59.248931 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.247611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-host-var-lib-cni-bin\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.248931 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.247618    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2060cba3-4269-4ecd-a03c-7a51fad57d0c-multus-daemon-config\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.248931 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.247679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-run-ovn\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.248931 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.247696    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-host-cni-netd\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.248931 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.247704    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-host-var-lib-kubelet\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.248931 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.247731    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-host-var-lib-cni-bin\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.248931 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.247772    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47ljz\" (UniqueName: \"kubernetes.io/projected/ffb473f1-6af6-40d5-b15b-23939d3eff20-kube-api-access-47ljz\") pod \"network-check-target-jlprw\" (UID: \"ffb473f1-6af6-40d5-b15b-23939d3eff20\") " pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:31:59.248931 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.247836    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2060cba3-4269-4ecd-a03c-7a51fad57d0c-host-var-lib-kubelet\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.248931 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.248008    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-run-ovn\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.249793 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.248087    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a649f06f-00a6-4ae6-8e68-ce87925329da-etc-tuned\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.249793 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.248180    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c5d1a464-df78-4f33-8ae7-f81183a9fb39-konnectivity-ca\") pod \"konnectivity-agent-xh2bl\" (UID: \"c5d1a464-df78-4f33-8ae7-f81183a9fb39\") " pod="kube-system/konnectivity-agent-xh2bl"
Jul 07 06:31:59.249793 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.249538    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c5d1a464-df78-4f33-8ae7-f81183a9fb39-agent-certs\") pod \"konnectivity-agent-xh2bl\" (UID: \"c5d1a464-df78-4f33-8ae7-f81183a9fb39\") " pod="kube-system/konnectivity-agent-xh2bl"
Jul 07 06:31:59.249793 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.249665    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a649f06f-00a6-4ae6-8e68-ce87925329da-tmp\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.250268 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.250244    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-ovn-node-metrics-cert\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.253311 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:59.252900    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 06:31:59.253311 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:59.252925    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 06:31:59.253311 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:59.252943    2563 projected.go:196] Error preparing data for projected volume kube-api-access-47ljz for pod openshift-network-diagnostics/network-check-target-jlprw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:31:59.253311 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:59.253009    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffb473f1-6af6-40d5-b15b-23939d3eff20-kube-api-access-47ljz podName:ffb473f1-6af6-40d5-b15b-23939d3eff20 nodeName:}" failed. No retries permitted until 2026-07-07 06:31:59.752989348 +0000 UTC m=+3.117185888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-47ljz" (UniqueName: "kubernetes.io/projected/ffb473f1-6af6-40d5-b15b-23939d3eff20-kube-api-access-47ljz") pod "network-check-target-jlprw" (UID: "ffb473f1-6af6-40d5-b15b-23939d3eff20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:31:59.255868 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.255781    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7qpw\" (UniqueName: \"kubernetes.io/projected/2060cba3-4269-4ecd-a03c-7a51fad57d0c-kube-api-access-v7qpw\") pod \"multus-54wzm\" (UID: \"2060cba3-4269-4ecd-a03c-7a51fad57d0c\") " pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.256909 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.256889    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m48ld\" (UniqueName: \"kubernetes.io/projected/6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e-kube-api-access-m48ld\") pod \"ovnkube-node-wnljf\" (UID: \"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e\") " pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.260050 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.260029    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6khjs\" (UniqueName: \"kubernetes.io/projected/a649f06f-00a6-4ae6-8e68-ce87925329da-kube-api-access-6khjs\") pod \"tuned-sc8n2\" (UID: \"a649f06f-00a6-4ae6-8e68-ce87925329da\") " pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.329193 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.329154    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 06:31:59.334790 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.334749    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kh9w8"
Jul 07 06:31:59.344022 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.343991    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6"
Jul 07 06:31:59.356366 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.356324    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mbqms"
Jul 07 06:31:59.359375 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.359350    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b6cs5"
Jul 07 06:31:59.368559 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.368524    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jg66b"
Jul 07 06:31:59.378584 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.378554    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xh2bl"
Jul 07 06:31:59.385725 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.385695    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:31:59.394658 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.394613    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sc8n2"
Jul 07 06:31:59.402885 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.402848    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-54wzm"
Jul 07 06:31:59.524951 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.524878    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 06:31:59.650345 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.650315    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee9fd62f-81cf-4544-8454-9fa9f53c0768-original-pull-secret\") pod \"global-pull-secret-syncer-57vh8\" (UID: \"ee9fd62f-81cf-4544-8454-9fa9f53c0768\") " pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:31:59.650512 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.650367    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c1e103c-f156-4b8a-8426-f18974595c51-metrics-certs\") pod \"network-metrics-daemon-mx8qb\" (UID: \"7c1e103c-f156-4b8a-8426-f18974595c51\") " pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:31:59.650512 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:59.650473    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:31:59.650512 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:59.650473    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 06:31:59.650612 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:59.650542    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c1e103c-f156-4b8a-8426-f18974595c51-metrics-certs podName:7c1e103c-f156-4b8a-8426-f18974595c51 nodeName:}" failed. No retries permitted until 2026-07-07 06:32:00.6505217 +0000 UTC m=+4.014718238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c1e103c-f156-4b8a-8426-f18974595c51-metrics-certs") pod "network-metrics-daemon-mx8qb" (UID: "7c1e103c-f156-4b8a-8426-f18974595c51") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:31:59.650612 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:59.650556    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee9fd62f-81cf-4544-8454-9fa9f53c0768-original-pull-secret podName:ee9fd62f-81cf-4544-8454-9fa9f53c0768 nodeName:}" failed. No retries permitted until 2026-07-07 06:32:00.650550217 +0000 UTC m=+4.014746739 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ee9fd62f-81cf-4544-8454-9fa9f53c0768-original-pull-secret") pod "global-pull-secret-syncer-57vh8" (UID: "ee9fd62f-81cf-4544-8454-9fa9f53c0768") : object "kube-system"/"original-pull-secret" not registered
Jul 07 06:31:59.851206 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:31:59.851166    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47ljz\" (UniqueName: \"kubernetes.io/projected/ffb473f1-6af6-40d5-b15b-23939d3eff20-kube-api-access-47ljz\") pod \"network-check-target-jlprw\" (UID: \"ffb473f1-6af6-40d5-b15b-23939d3eff20\") " pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:31:59.851387 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:59.851341    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 06:31:59.851387 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:59.851363    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 06:31:59.851387 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:59.851372    2563 projected.go:196] Error preparing data for projected volume kube-api-access-47ljz for pod openshift-network-diagnostics/network-check-target-jlprw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:31:59.851536 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:31:59.851449    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffb473f1-6af6-40d5-b15b-23939d3eff20-kube-api-access-47ljz podName:ffb473f1-6af6-40d5-b15b-23939d3eff20 nodeName:}" failed. No retries permitted until 2026-07-07 06:32:00.851429099 +0000 UTC m=+4.215625621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-47ljz" (UniqueName: "kubernetes.io/projected/ffb473f1-6af6-40d5-b15b-23939d3eff20-kube-api-access-47ljz") pod "network-check-target-jlprw" (UID: "ffb473f1-6af6-40d5-b15b-23939d3eff20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:32:00.068355 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:00.068263    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 06:26:59 +0000 UTC" deadline="2028-01-27 16:44:26.017412307 +0000 UTC"
Jul 07 06:32:00.068355 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:00.068300    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13666h12m25.949115768s"
Jul 07 06:32:00.353971 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:32:00.353933    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce422fc1_946c_45e5_b5bf_2203adc5f7f9.slice/crio-f83c7cc2cfa5f117c4c95687b80ee40712bcd5c276ec2964a2085c54cb144b87 WatchSource:0}: Error finding container f83c7cc2cfa5f117c4c95687b80ee40712bcd5c276ec2964a2085c54cb144b87: Status 404 returned error can't find the container with id f83c7cc2cfa5f117c4c95687b80ee40712bcd5c276ec2964a2085c54cb144b87
Jul 07 06:32:00.355961 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:32:00.355849    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2060cba3_4269_4ecd_a03c_7a51fad57d0c.slice/crio-331dda7944402ea9bfe84c05c14dbc790d1eff7f2bdf2fe2908db7728d935657 WatchSource:0}: Error finding container 331dda7944402ea9bfe84c05c14dbc790d1eff7f2bdf2fe2908db7728d935657: Status 404 returned error can't find the container with id 331dda7944402ea9bfe84c05c14dbc790d1eff7f2bdf2fe2908db7728d935657
Jul 07 06:32:00.359817 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:32:00.359733    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod327fd7db_21be_45ef_9efd_6c3f46c1e663.slice/crio-bb0f2b91ce0d038f7f5242138a4b6fc60da3ed9155df95d52bbf689b2997627e WatchSource:0}: Error finding container bb0f2b91ce0d038f7f5242138a4b6fc60da3ed9155df95d52bbf689b2997627e: Status 404 returned error can't find the container with id bb0f2b91ce0d038f7f5242138a4b6fc60da3ed9155df95d52bbf689b2997627e
Jul 07 06:32:00.360400 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:32:00.360370    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d1a464_df78_4f33_8ae7_f81183a9fb39.slice/crio-c3b8d054375c76b6a2892585e2f07b9146459fc4b6e92a8eecf644acb9022a54 WatchSource:0}: Error finding container c3b8d054375c76b6a2892585e2f07b9146459fc4b6e92a8eecf644acb9022a54: Status 404 returned error can't find the container with id c3b8d054375c76b6a2892585e2f07b9146459fc4b6e92a8eecf644acb9022a54
Jul 07 06:32:00.364464 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:32:00.363730    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda649f06f_00a6_4ae6_8e68_ce87925329da.slice/crio-d1e1a8cd8d5e0bcd03590aec5ccf3346a03f24e7bc0f137b3dce51f23a2a5582 WatchSource:0}: Error finding container d1e1a8cd8d5e0bcd03590aec5ccf3346a03f24e7bc0f137b3dce51f23a2a5582: Status 404 returned error can't find the container with id d1e1a8cd8d5e0bcd03590aec5ccf3346a03f24e7bc0f137b3dce51f23a2a5582
Jul 07 06:32:00.370970 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:32:00.370876    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eedcc8c_8c57_4b1a_84b2_4cf9d9efd52e.slice/crio-b59e1dcad632b52d86c4bf6dbdc360dcaa50f410884d878b0b0fb6f8137a0917 WatchSource:0}: Error finding container b59e1dcad632b52d86c4bf6dbdc360dcaa50f410884d878b0b0fb6f8137a0917: Status 404 returned error can't find the container with id b59e1dcad632b52d86c4bf6dbdc360dcaa50f410884d878b0b0fb6f8137a0917
Jul 07 06:32:00.658663 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:00.656913    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee9fd62f-81cf-4544-8454-9fa9f53c0768-original-pull-secret\") pod \"global-pull-secret-syncer-57vh8\" (UID: \"ee9fd62f-81cf-4544-8454-9fa9f53c0768\") " pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:00.658663 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:00.657144    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c1e103c-f156-4b8a-8426-f18974595c51-metrics-certs\") pod \"network-metrics-daemon-mx8qb\" (UID: \"7c1e103c-f156-4b8a-8426-f18974595c51\") " pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:00.658663 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:00.657075    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 06:32:00.658663 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:00.657241    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:32:00.658663 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:00.657253    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee9fd62f-81cf-4544-8454-9fa9f53c0768-original-pull-secret podName:ee9fd62f-81cf-4544-8454-9fa9f53c0768 nodeName:}" failed. No retries permitted until 2026-07-07 06:32:02.657232577 +0000 UTC m=+6.021429104 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ee9fd62f-81cf-4544-8454-9fa9f53c0768-original-pull-secret") pod "global-pull-secret-syncer-57vh8" (UID: "ee9fd62f-81cf-4544-8454-9fa9f53c0768") : object "kube-system"/"original-pull-secret" not registered
Jul 07 06:32:00.658663 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:00.657283    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c1e103c-f156-4b8a-8426-f18974595c51-metrics-certs podName:7c1e103c-f156-4b8a-8426-f18974595c51 nodeName:}" failed. No retries permitted until 2026-07-07 06:32:02.657268983 +0000 UTC m=+6.021465552 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c1e103c-f156-4b8a-8426-f18974595c51-metrics-certs") pod "network-metrics-daemon-mx8qb" (UID: "7c1e103c-f156-4b8a-8426-f18974595c51") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:32:00.859237 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:00.859147    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47ljz\" (UniqueName: \"kubernetes.io/projected/ffb473f1-6af6-40d5-b15b-23939d3eff20-kube-api-access-47ljz\") pod \"network-check-target-jlprw\" (UID: \"ffb473f1-6af6-40d5-b15b-23939d3eff20\") " pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:00.859372 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:00.859343    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 06:32:00.859372 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:00.859371    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 06:32:00.859445 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:00.859385    2563 projected.go:196] Error preparing data for projected volume kube-api-access-47ljz for pod openshift-network-diagnostics/network-check-target-jlprw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:32:00.859477 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:00.859449    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffb473f1-6af6-40d5-b15b-23939d3eff20-kube-api-access-47ljz podName:ffb473f1-6af6-40d5-b15b-23939d3eff20 nodeName:}" failed. No retries permitted until 2026-07-07 06:32:02.859431047 +0000 UTC m=+6.223627574 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-47ljz" (UniqueName: "kubernetes.io/projected/ffb473f1-6af6-40d5-b15b-23939d3eff20-kube-api-access-47ljz") pod "network-check-target-jlprw" (UID: "ffb473f1-6af6-40d5-b15b-23939d3eff20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:32:01.068524 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:01.068403    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 06:26:59 +0000 UTC" deadline="2028-02-16 02:22:05.193912324 +0000 UTC"
Jul 07 06:32:01.068524 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:01.068472    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14131h50m4.125443797s"
Jul 07 06:32:01.166755 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:01.166174    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:01.166755 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:01.166203    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:01.166755 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:01.166316    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlprw" podUID="ffb473f1-6af6-40d5-b15b-23939d3eff20"
Jul 07 06:32:01.166755 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:01.166381    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx8qb" podUID="7c1e103c-f156-4b8a-8426-f18974595c51"
Jul 07 06:32:01.166755 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:01.166442    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:01.166755 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:01.166511    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57vh8" podUID="ee9fd62f-81cf-4544-8454-9fa9f53c0768"
Jul 07 06:32:01.196961 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:01.196919    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-92.ec2.internal" event={"ID":"9ed9ea7211a1e46135e9e4a3c3edc4de","Type":"ContainerStarted","Data":"72de31bab973576ccbd1f50f372c9e394acd75aadf46f7d783db9d791361ae49"}
Jul 07 06:32:01.219906 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:01.219716    2563 generic.go:356] "Generic (PLEG): container finished" podID="10974f874b8e23ee02e9135453ab4de0" containerID="7f3f17e1e17216912c9cccb6d0dc5b92f49ad257593ac55a72c7e745e3e0d158" exitCode=0
Jul 07 06:32:01.219906 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:01.219821    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-92.ec2.internal" event={"ID":"10974f874b8e23ee02e9135453ab4de0","Type":"ContainerDied","Data":"7f3f17e1e17216912c9cccb6d0dc5b92f49ad257593ac55a72c7e745e3e0d158"}
Jul 07 06:32:01.235112 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:01.235073    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b6cs5" event={"ID":"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c","Type":"ContainerStarted","Data":"8dea7e9478cb52101549156dd7bfc3a4c43db4b7482221b650384ca7e1bd75e2"}
Jul 07 06:32:01.238006 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:01.237933    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-92.ec2.internal" podStartSLOduration=3.237912883 podStartE2EDuration="3.237912883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 06:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:32:01.212156841 +0000 UTC m=+4.576353386" watchObservedRunningTime="2026-07-07 06:32:01.237912883 +0000 UTC m=+4.602109429"
Jul 07 06:32:01.262499 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:01.262457    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sc8n2" event={"ID":"a649f06f-00a6-4ae6-8e68-ce87925329da","Type":"ContainerStarted","Data":"d1e1a8cd8d5e0bcd03590aec5ccf3346a03f24e7bc0f137b3dce51f23a2a5582"}
Jul 07 06:32:01.277466 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:01.277430    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kh9w8" event={"ID":"1a9cb87a-eb89-462d-b9dc-83e0f39a712d","Type":"ContainerStarted","Data":"d0f8d30f2374c24921782f78e4193ae638d0a9fd13d73be1b5f4e4b3f046c3b4"}
Jul 07 06:32:01.283938 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:01.283894    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mbqms" event={"ID":"4aaa39b2-a25e-4a3c-b3b8-b65d0b6d140b","Type":"ContainerStarted","Data":"d07e6094348890a6c885b8e513b0a10bca897ba3377723ac8bd4f1015556fac8"}
Jul 07 06:32:01.303488 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:01.303449    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-54wzm" event={"ID":"2060cba3-4269-4ecd-a03c-7a51fad57d0c","Type":"ContainerStarted","Data":"331dda7944402ea9bfe84c05c14dbc790d1eff7f2bdf2fe2908db7728d935657"}
Jul 07 06:32:01.312747 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:01.312707    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wnljf" event={"ID":"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e","Type":"ContainerStarted","Data":"b59e1dcad632b52d86c4bf6dbdc360dcaa50f410884d878b0b0fb6f8137a0917"}
Jul 07 06:32:01.329605 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:01.329560    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xh2bl" event={"ID":"c5d1a464-df78-4f33-8ae7-f81183a9fb39","Type":"ContainerStarted","Data":"c3b8d054375c76b6a2892585e2f07b9146459fc4b6e92a8eecf644acb9022a54"}
Jul 07 06:32:01.335471 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:01.334894    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jg66b" event={"ID":"327fd7db-21be-45ef-9efd-6c3f46c1e663","Type":"ContainerStarted","Data":"bb0f2b91ce0d038f7f5242138a4b6fc60da3ed9155df95d52bbf689b2997627e"}
Jul 07 06:32:01.337606 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:01.337564    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6" event={"ID":"ce422fc1-946c-45e5-b5bf-2203adc5f7f9","Type":"ContainerStarted","Data":"f83c7cc2cfa5f117c4c95687b80ee40712bcd5c276ec2964a2085c54cb144b87"}
Jul 07 06:32:02.349289 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:02.349239    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-92.ec2.internal" event={"ID":"10974f874b8e23ee02e9135453ab4de0","Type":"ContainerStarted","Data":"fd536aba24588dfe7d16dc95754700f81a092a83bb86eb7072f82b0b3dbbccfe"}
Jul 07 06:32:02.678018 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:02.677912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c1e103c-f156-4b8a-8426-f18974595c51-metrics-certs\") pod \"network-metrics-daemon-mx8qb\" (UID: \"7c1e103c-f156-4b8a-8426-f18974595c51\") " pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:02.678191 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:02.678032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee9fd62f-81cf-4544-8454-9fa9f53c0768-original-pull-secret\") pod \"global-pull-secret-syncer-57vh8\" (UID: \"ee9fd62f-81cf-4544-8454-9fa9f53c0768\") " pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:02.678191 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:02.678174    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 06:32:02.678304 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:02.678235    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee9fd62f-81cf-4544-8454-9fa9f53c0768-original-pull-secret podName:ee9fd62f-81cf-4544-8454-9fa9f53c0768 nodeName:}" failed. No retries permitted until 2026-07-07 06:32:06.678217534 +0000 UTC m=+10.042414071 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ee9fd62f-81cf-4544-8454-9fa9f53c0768-original-pull-secret") pod "global-pull-secret-syncer-57vh8" (UID: "ee9fd62f-81cf-4544-8454-9fa9f53c0768") : object "kube-system"/"original-pull-secret" not registered
Jul 07 06:32:02.678704 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:02.678654    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:32:02.678803 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:02.678709    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c1e103c-f156-4b8a-8426-f18974595c51-metrics-certs podName:7c1e103c-f156-4b8a-8426-f18974595c51 nodeName:}" failed. No retries permitted until 2026-07-07 06:32:06.678692142 +0000 UTC m=+10.042888665 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c1e103c-f156-4b8a-8426-f18974595c51-metrics-certs") pod "network-metrics-daemon-mx8qb" (UID: "7c1e103c-f156-4b8a-8426-f18974595c51") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:32:02.879628 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:02.879573    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47ljz\" (UniqueName: \"kubernetes.io/projected/ffb473f1-6af6-40d5-b15b-23939d3eff20-kube-api-access-47ljz\") pod \"network-check-target-jlprw\" (UID: \"ffb473f1-6af6-40d5-b15b-23939d3eff20\") " pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:02.879909 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:02.879887    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 06:32:02.879982 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:02.879916    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 06:32:02.879982 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:02.879930    2563 projected.go:196] Error preparing data for projected volume kube-api-access-47ljz for pod openshift-network-diagnostics/network-check-target-jlprw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:32:02.880096 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:02.879992    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffb473f1-6af6-40d5-b15b-23939d3eff20-kube-api-access-47ljz podName:ffb473f1-6af6-40d5-b15b-23939d3eff20 nodeName:}" failed. No retries permitted until 2026-07-07 06:32:06.87997173 +0000 UTC m=+10.244168255 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-47ljz" (UniqueName: "kubernetes.io/projected/ffb473f1-6af6-40d5-b15b-23939d3eff20-kube-api-access-47ljz") pod "network-check-target-jlprw" (UID: "ffb473f1-6af6-40d5-b15b-23939d3eff20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:32:03.167280 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:03.166747    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:03.167280 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:03.166764    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:03.167280 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:03.166880    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlprw" podUID="ffb473f1-6af6-40d5-b15b-23939d3eff20"
Jul 07 06:32:03.167280 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:03.166959    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx8qb" podUID="7c1e103c-f156-4b8a-8426-f18974595c51"
Jul 07 06:32:03.167280 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:03.167027    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:03.167280 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:03.167109    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57vh8" podUID="ee9fd62f-81cf-4544-8454-9fa9f53c0768"
Jul 07 06:32:05.165607 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:05.165533    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:05.165607 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:05.165556    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:05.166132 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:05.165682    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlprw" podUID="ffb473f1-6af6-40d5-b15b-23939d3eff20"
Jul 07 06:32:05.166132 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:05.165759    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx8qb" podUID="7c1e103c-f156-4b8a-8426-f18974595c51"
Jul 07 06:32:05.166132 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:05.165809    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:05.166132 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:05.165892    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57vh8" podUID="ee9fd62f-81cf-4544-8454-9fa9f53c0768"
Jul 07 06:32:06.712941 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:06.712897    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee9fd62f-81cf-4544-8454-9fa9f53c0768-original-pull-secret\") pod \"global-pull-secret-syncer-57vh8\" (UID: \"ee9fd62f-81cf-4544-8454-9fa9f53c0768\") " pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:06.713364 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:06.712968    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c1e103c-f156-4b8a-8426-f18974595c51-metrics-certs\") pod \"network-metrics-daemon-mx8qb\" (UID: \"7c1e103c-f156-4b8a-8426-f18974595c51\") " pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:06.713364 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:06.713052    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:32:06.713364 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:06.713052    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 06:32:06.713364 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:06.713122    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c1e103c-f156-4b8a-8426-f18974595c51-metrics-certs podName:7c1e103c-f156-4b8a-8426-f18974595c51 nodeName:}" failed. No retries permitted until 2026-07-07 06:32:14.713100704 +0000 UTC m=+18.077297247 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c1e103c-f156-4b8a-8426-f18974595c51-metrics-certs") pod "network-metrics-daemon-mx8qb" (UID: "7c1e103c-f156-4b8a-8426-f18974595c51") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:32:06.713364 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:06.713141    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee9fd62f-81cf-4544-8454-9fa9f53c0768-original-pull-secret podName:ee9fd62f-81cf-4544-8454-9fa9f53c0768 nodeName:}" failed. No retries permitted until 2026-07-07 06:32:14.713131383 +0000 UTC m=+18.077327910 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ee9fd62f-81cf-4544-8454-9fa9f53c0768-original-pull-secret") pod "global-pull-secret-syncer-57vh8" (UID: "ee9fd62f-81cf-4544-8454-9fa9f53c0768") : object "kube-system"/"original-pull-secret" not registered
Jul 07 06:32:06.915351 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:06.915306    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47ljz\" (UniqueName: \"kubernetes.io/projected/ffb473f1-6af6-40d5-b15b-23939d3eff20-kube-api-access-47ljz\") pod \"network-check-target-jlprw\" (UID: \"ffb473f1-6af6-40d5-b15b-23939d3eff20\") " pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:06.915549 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:06.915506    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 06:32:06.915549 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:06.915531    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 06:32:06.915549 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:06.915543    2563 projected.go:196] Error preparing data for projected volume kube-api-access-47ljz for pod openshift-network-diagnostics/network-check-target-jlprw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:32:06.915741 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:06.915611    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffb473f1-6af6-40d5-b15b-23939d3eff20-kube-api-access-47ljz podName:ffb473f1-6af6-40d5-b15b-23939d3eff20 nodeName:}" failed. No retries permitted until 2026-07-07 06:32:14.915591872 +0000 UTC m=+18.279788408 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-47ljz" (UniqueName: "kubernetes.io/projected/ffb473f1-6af6-40d5-b15b-23939d3eff20-kube-api-access-47ljz") pod "network-check-target-jlprw" (UID: "ffb473f1-6af6-40d5-b15b-23939d3eff20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:32:07.167740 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:07.167664    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:07.167891 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:07.167798    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx8qb" podUID="7c1e103c-f156-4b8a-8426-f18974595c51"
Jul 07 06:32:07.168119 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:07.168101    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:07.168218 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:07.168193    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57vh8" podUID="ee9fd62f-81cf-4544-8454-9fa9f53c0768"
Jul 07 06:32:07.168287 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:07.168260    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:07.168346 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:07.168329    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlprw" podUID="ffb473f1-6af6-40d5-b15b-23939d3eff20"
Jul 07 06:32:09.167361 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:09.166734    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:09.167361 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:09.166843    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57vh8" podUID="ee9fd62f-81cf-4544-8454-9fa9f53c0768"
Jul 07 06:32:09.167361 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:09.167217    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:09.167361 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:09.167309    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlprw" podUID="ffb473f1-6af6-40d5-b15b-23939d3eff20"
Jul 07 06:32:09.168048 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:09.167450    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:09.168048 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:09.167523    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx8qb" podUID="7c1e103c-f156-4b8a-8426-f18974595c51"
Jul 07 06:32:11.166262 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:11.166221    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:11.166701 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:11.166221    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:11.166701 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:11.166386    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:11.166701 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:11.166381    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx8qb" podUID="7c1e103c-f156-4b8a-8426-f18974595c51"
Jul 07 06:32:11.166701 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:11.166469    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlprw" podUID="ffb473f1-6af6-40d5-b15b-23939d3eff20"
Jul 07 06:32:11.166701 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:11.166556    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57vh8" podUID="ee9fd62f-81cf-4544-8454-9fa9f53c0768"
Jul 07 06:32:13.165907 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:13.165870    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:13.166401 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:13.166007    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57vh8" podUID="ee9fd62f-81cf-4544-8454-9fa9f53c0768"
Jul 07 06:32:13.166401 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:13.165870    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:13.166401 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:13.166115    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlprw" podUID="ffb473f1-6af6-40d5-b15b-23939d3eff20"
Jul 07 06:32:13.166401 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:13.165871    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:13.166401 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:13.166201    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx8qb" podUID="7c1e103c-f156-4b8a-8426-f18974595c51"
Jul 07 06:32:14.777682 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:14.777617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee9fd62f-81cf-4544-8454-9fa9f53c0768-original-pull-secret\") pod \"global-pull-secret-syncer-57vh8\" (UID: \"ee9fd62f-81cf-4544-8454-9fa9f53c0768\") " pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:14.778158 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:14.777712    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c1e103c-f156-4b8a-8426-f18974595c51-metrics-certs\") pod \"network-metrics-daemon-mx8qb\" (UID: \"7c1e103c-f156-4b8a-8426-f18974595c51\") " pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:14.778158 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:14.777786    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 06:32:14.778158 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:14.777850    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee9fd62f-81cf-4544-8454-9fa9f53c0768-original-pull-secret podName:ee9fd62f-81cf-4544-8454-9fa9f53c0768 nodeName:}" failed. No retries permitted until 2026-07-07 06:32:30.777832958 +0000 UTC m=+34.142029481 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ee9fd62f-81cf-4544-8454-9fa9f53c0768-original-pull-secret") pod "global-pull-secret-syncer-57vh8" (UID: "ee9fd62f-81cf-4544-8454-9fa9f53c0768") : object "kube-system"/"original-pull-secret" not registered
Jul 07 06:32:14.778158 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:14.777872    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:32:14.778158 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:14.777923    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c1e103c-f156-4b8a-8426-f18974595c51-metrics-certs podName:7c1e103c-f156-4b8a-8426-f18974595c51 nodeName:}" failed. No retries permitted until 2026-07-07 06:32:30.777908878 +0000 UTC m=+34.142105401 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c1e103c-f156-4b8a-8426-f18974595c51-metrics-certs") pod "network-metrics-daemon-mx8qb" (UID: "7c1e103c-f156-4b8a-8426-f18974595c51") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:32:14.980259 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:14.980224    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47ljz\" (UniqueName: \"kubernetes.io/projected/ffb473f1-6af6-40d5-b15b-23939d3eff20-kube-api-access-47ljz\") pod \"network-check-target-jlprw\" (UID: \"ffb473f1-6af6-40d5-b15b-23939d3eff20\") " pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:14.980466 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:14.980428    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 06:32:14.980466 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:14.980468    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 06:32:14.980650 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:14.980484    2563 projected.go:196] Error preparing data for projected volume kube-api-access-47ljz for pod openshift-network-diagnostics/network-check-target-jlprw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:32:14.980650 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:14.980549    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffb473f1-6af6-40d5-b15b-23939d3eff20-kube-api-access-47ljz podName:ffb473f1-6af6-40d5-b15b-23939d3eff20 nodeName:}" failed. No retries permitted until 2026-07-07 06:32:30.980534103 +0000 UTC m=+34.344730626 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-47ljz" (UniqueName: "kubernetes.io/projected/ffb473f1-6af6-40d5-b15b-23939d3eff20-kube-api-access-47ljz") pod "network-check-target-jlprw" (UID: "ffb473f1-6af6-40d5-b15b-23939d3eff20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:32:15.166088 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:15.166056    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:15.166279 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:15.166056    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:15.166279 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:15.166185    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlprw" podUID="ffb473f1-6af6-40d5-b15b-23939d3eff20"
Jul 07 06:32:15.166279 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:15.166056    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:15.166279 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:15.166260    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx8qb" podUID="7c1e103c-f156-4b8a-8426-f18974595c51"
Jul 07 06:32:15.166478 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:15.166339    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57vh8" podUID="ee9fd62f-81cf-4544-8454-9fa9f53c0768"
Jul 07 06:32:17.167492 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:17.167452    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:17.167980 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:17.167545    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:17.167980 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:17.167574    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:17.167980 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:17.167616    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57vh8" podUID="ee9fd62f-81cf-4544-8454-9fa9f53c0768"
Jul 07 06:32:17.167980 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:17.167663    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlprw" podUID="ffb473f1-6af6-40d5-b15b-23939d3eff20"
Jul 07 06:32:17.167980 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:17.167730    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx8qb" podUID="7c1e103c-f156-4b8a-8426-f18974595c51"
Jul 07 06:32:19.166313 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:19.166270    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:19.166917 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:19.166269    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:19.166917 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:19.166414    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57vh8" podUID="ee9fd62f-81cf-4544-8454-9fa9f53c0768"
Jul 07 06:32:19.166917 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:19.166510    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx8qb" podUID="7c1e103c-f156-4b8a-8426-f18974595c51"
Jul 07 06:32:19.166917 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:19.166269    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:19.166917 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:19.166611    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlprw" podUID="ffb473f1-6af6-40d5-b15b-23939d3eff20"
Jul 07 06:32:20.397765 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:20.397303    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wnljf" event={"ID":"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e","Type":"ContainerStarted","Data":"e2e13b374b9d4b681cef7c124a25782dc44e37b71e7d649a0b8f2827d7a06640"}
Jul 07 06:32:20.398623 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:20.397856    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wnljf" event={"ID":"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e","Type":"ContainerStarted","Data":"48ac76821b680a963a4d9f8db8f57105e6ab39bfb716b8f7f16e9572a1e8c3a2"}
Jul 07 06:32:20.398623 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:20.397914    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wnljf" event={"ID":"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e","Type":"ContainerStarted","Data":"04fd4d46f441459a5b516d5ee35ecc26399479c248e1ac01131738ee48288ea3"}
Jul 07 06:32:20.398623 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:20.397957    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wnljf" event={"ID":"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e","Type":"ContainerStarted","Data":"be50ef8e3899cf67be8050bf9a44fd118dabe21fc66bf5cd65d89e1a5f5a7127"}
Jul 07 06:32:20.398623 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:20.397999    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wnljf" event={"ID":"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e","Type":"ContainerStarted","Data":"bbd391136cbd65527934231c358b8c4333d17914224e124c3e1ac359fa02465d"}
Jul 07 06:32:20.398623 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:20.398041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wnljf" event={"ID":"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e","Type":"ContainerStarted","Data":"a9c6045e6b2f5ba2fb9cf06ba092f64f16b61ce96b55f0c277911ca28b6374ca"}
Jul 07 06:32:20.398954 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:20.398918    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xh2bl" event={"ID":"c5d1a464-df78-4f33-8ae7-f81183a9fb39","Type":"ContainerStarted","Data":"47a21e8ef89a5f75503c801e463bc34c21c8a9e8b67375c1256d63c7a0eb4f15"}
Jul 07 06:32:20.400540 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:20.400511    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jg66b" event={"ID":"327fd7db-21be-45ef-9efd-6c3f46c1e663","Type":"ContainerStarted","Data":"6c090a3fcd6683e2a1846aaedda448adf69d777d8459f2f97cd62b29b6d95bdd"}
Jul 07 06:32:20.402300 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:20.402268    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6" event={"ID":"ce422fc1-946c-45e5-b5bf-2203adc5f7f9","Type":"ContainerStarted","Data":"399ff19bb26a8b7a4d1d621903f9116ea9046ea7b9e8cdbeda6468780243df9b"}
Jul 07 06:32:20.403901 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:20.403880    2563 generic.go:356] "Generic (PLEG): container finished" podID="0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c" containerID="352f7f5dd4eddc420cb0e2ace5c47a4745f1041ad4f7bd6b590c1d48b230db26" exitCode=0
Jul 07 06:32:20.403998 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:20.403949    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b6cs5" event={"ID":"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c","Type":"ContainerDied","Data":"352f7f5dd4eddc420cb0e2ace5c47a4745f1041ad4f7bd6b590c1d48b230db26"}
Jul 07 06:32:20.405438 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:20.405418    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sc8n2" event={"ID":"a649f06f-00a6-4ae6-8e68-ce87925329da","Type":"ContainerStarted","Data":"fef750acc16c615f0ed6dd6fc3604f58d0348a7247d993159f9f0c1f64b7d254"}
Jul 07 06:32:20.407535 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:20.407510    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mbqms" event={"ID":"4aaa39b2-a25e-4a3c-b3b8-b65d0b6d140b","Type":"ContainerStarted","Data":"a9cf62dba472f98ee70353b0ff671d73c88c7a0f4da13de9afcab3f33354e80c"}
Jul 07 06:32:20.409079 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:20.409050    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-54wzm" event={"ID":"2060cba3-4269-4ecd-a03c-7a51fad57d0c","Type":"ContainerStarted","Data":"a2737c9e3b218e62402258ca14e21beff21f69edc216badf62a7f4676e9b0bfb"}
Jul 07 06:32:20.417624 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:20.417574    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-92.ec2.internal" podStartSLOduration=22.417556325 podStartE2EDuration="22.417556325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 06:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:32:02.377742182 +0000 UTC m=+5.741938728" watchObservedRunningTime="2026-07-07 06:32:20.417556325 +0000 UTC m=+23.781752870"
Jul 07 06:32:20.417975 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:20.417947    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xh2bl" podStartSLOduration=12.274192576 podStartE2EDuration="23.417938002s" totalImagesPullingTime="11.143745426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:32:20.417427138 +0000 UTC m=+23.781623695" watchObservedRunningTime="2026-07-07 06:32:20.417938002 +0000 UTC m=+23.782134546"
Jul 07 06:32:20.432870 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:20.432816    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mbqms" podStartSLOduration=4.315469381 podStartE2EDuration="23.432797418s" totalImagesPullingTime="19.117328037s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 06:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:32:20.432454168 +0000 UTC m=+23.796650714" watchObservedRunningTime="2026-07-07 06:32:20.432797418 +0000 UTC m=+23.796993965"
Jul 07 06:32:20.475780 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:20.475716    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jg66b" podStartSLOduration=4.348939732 podStartE2EDuration="23.475695359s" totalImagesPullingTime="19.126755627s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 06:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:32:20.450986221 +0000 UTC m=+23.815182767" watchObservedRunningTime="2026-07-07 06:32:20.475695359 +0000 UTC m=+23.839891899"
Jul 07 06:32:20.524461 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:20.524409    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sc8n2" podStartSLOduration=4.401958211 podStartE2EDuration="23.524392722s" totalImagesPullingTime="19.122434511s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 06:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:32:20.498596851 +0000 UTC m=+23.862793395" watchObservedRunningTime="2026-07-07 06:32:20.524392722 +0000 UTC m=+23.888589267"
Jul 07 06:32:20.524663 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:20.524572    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-54wzm" podStartSLOduration=4.315760153 podStartE2EDuration="23.524568255s" totalImagesPullingTime="19.208808102s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 06:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:32:20.524113412 +0000 UTC m=+23.888309957" watchObservedRunningTime="2026-07-07 06:32:20.524568255 +0000 UTC m=+23.888764801"
Jul 07 06:32:21.165714 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:21.165687    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:21.165714 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:21.165703    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:21.165977 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:21.165688    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:21.165977 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:21.165826    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlprw" podUID="ffb473f1-6af6-40d5-b15b-23939d3eff20"
Jul 07 06:32:21.165977 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:21.165875    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57vh8" podUID="ee9fd62f-81cf-4544-8454-9fa9f53c0768"
Jul 07 06:32:21.166131 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:21.165956    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx8qb" podUID="7c1e103c-f156-4b8a-8426-f18974595c51"
Jul 07 06:32:21.205997 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:21.205958    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 07 06:32:21.412826 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:21.412732    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6" event={"ID":"ce422fc1-946c-45e5-b5bf-2203adc5f7f9","Type":"ContainerStarted","Data":"f6573f05054aee470a671def38a3ce19fe4833207606fc6f0672abf809afac82"}
Jul 07 06:32:21.414232 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:21.414195    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kh9w8" event={"ID":"1a9cb87a-eb89-462d-b9dc-83e0f39a712d","Type":"ContainerStarted","Data":"555348b325f5aabc9aa92df24f2b57222fe9eda49837fa9bd4dd9c3db1f471f9"}
Jul 07 06:32:22.102991 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:22.102872    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T06:32:21.205981702Z","UUID":"94b26780-8032-4b2c-8d61-83596ed9a56d","Handler":null,"Name":"","Endpoint":""}
Jul 07 06:32:22.104820 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:22.104796    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 07 06:32:22.104960 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:22.104841    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 07 06:32:22.420389 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:22.420306    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wnljf" event={"ID":"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e","Type":"ContainerStarted","Data":"108796a62bf90085fec51f531fecb0c4c00794c11adba7079dd60237880ee532"}
Jul 07 06:32:23.165542 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:23.165510    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:23.165752 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:23.165620    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:23.165752 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:23.165652    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57vh8" podUID="ee9fd62f-81cf-4544-8454-9fa9f53c0768"
Jul 07 06:32:23.165752 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:23.165738    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlprw" podUID="ffb473f1-6af6-40d5-b15b-23939d3eff20"
Jul 07 06:32:23.165914 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:23.165809    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:23.165914 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:23.165884    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx8qb" podUID="7c1e103c-f156-4b8a-8426-f18974595c51"
Jul 07 06:32:23.425728 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:23.425166    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6" event={"ID":"ce422fc1-946c-45e5-b5bf-2203adc5f7f9","Type":"ContainerStarted","Data":"1734e2decd19a4e4e5275c168a2c283ecbfe9ebf90bc42993b50a69002dcb4b4"}
Jul 07 06:32:23.447994 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:23.447932    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqlp6" podStartSLOduration=4.64414692 podStartE2EDuration="26.447915201s" totalImagesPullingTime="21.803768281s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 06:31:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:32:23.447722048 +0000 UTC m=+26.811918594" watchObservedRunningTime="2026-07-07 06:32:23.447915201 +0000 UTC m=+26.812111745"
Jul 07 06:32:23.448539 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:23.448501    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kh9w8" podStartSLOduration=7.326755782 podStartE2EDuration="26.448492131s" totalImagesPullingTime="19.121736349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 06:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:32:21.431688498 +0000 UTC m=+24.795885053" watchObservedRunningTime="2026-07-07 06:32:23.448492131 +0000 UTC m=+26.812688756"
Jul 07 06:32:24.591014 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:24.590972    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xh2bl"
Jul 07 06:32:24.591782 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:24.591755    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xh2bl"
Jul 07 06:32:25.165656 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:25.165553    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:25.165812 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:25.165553    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:25.165812 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:25.165697    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlprw" podUID="ffb473f1-6af6-40d5-b15b-23939d3eff20"
Jul 07 06:32:25.165812 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:25.165553    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:25.165812 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:25.165775    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57vh8" podUID="ee9fd62f-81cf-4544-8454-9fa9f53c0768"
Jul 07 06:32:25.166024 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:25.165823    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx8qb" podUID="7c1e103c-f156-4b8a-8426-f18974595c51"
Jul 07 06:32:25.429772 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:25.429688    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xh2bl"
Jul 07 06:32:25.430291 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:25.430275    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xh2bl"
Jul 07 06:32:26.435424 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:26.435167    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wnljf" event={"ID":"6eedcc8c-8c57-4b1a-84b2-4cf9d9efd52e","Type":"ContainerStarted","Data":"5bc7cf74f64da4aa838b153776266ebd6ca95636f94f86dd77454f4a14938975"}
Jul 07 06:32:26.461308 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:26.461262    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wnljf" podStartSLOduration=10.272309855 podStartE2EDuration="29.461246173s" totalImagesPullingTime="19.188936318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 06:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:32:26.460503201 +0000 UTC m=+29.824699756" watchObservedRunningTime="2026-07-07 06:32:26.461246173 +0000 UTC m=+29.825442718"
Jul 07 06:32:27.166415 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:27.166382    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:27.166648 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:27.166484    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlprw" podUID="ffb473f1-6af6-40d5-b15b-23939d3eff20"
Jul 07 06:32:27.166648 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:27.166512    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:27.166648 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:27.166560    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:27.166754 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:27.166654    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57vh8" podUID="ee9fd62f-81cf-4544-8454-9fa9f53c0768"
Jul 07 06:32:27.166754 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:27.166741    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx8qb" podUID="7c1e103c-f156-4b8a-8426-f18974595c51"
Jul 07 06:32:27.439772 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:27.439689    2563 generic.go:356] "Generic (PLEG): container finished" podID="0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c" containerID="99a3ece03b3df6bc0d926b46318d0c0f509bc1947664a4a3bc9f1480b6a171f1" exitCode=0
Jul 07 06:32:27.440226 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:27.439770    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b6cs5" event={"ID":"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c","Type":"ContainerDied","Data":"99a3ece03b3df6bc0d926b46318d0c0f509bc1947664a4a3bc9f1480b6a171f1"}
Jul 07 06:32:27.440226 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:27.440209    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:32:27.440338 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:27.440270    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:32:27.440338 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:27.440319    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:32:27.456685 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:27.456661    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:32:27.456807 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:27.456763    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:32:29.166115 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:29.166072    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:29.166542 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:29.166072    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:29.166542 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:29.166194    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx8qb" podUID="7c1e103c-f156-4b8a-8426-f18974595c51"
Jul 07 06:32:29.166542 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:29.166291    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57vh8" podUID="ee9fd62f-81cf-4544-8454-9fa9f53c0768"
Jul 07 06:32:29.166542 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:29.166072    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:29.166542 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:29.166384    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlprw" podUID="ffb473f1-6af6-40d5-b15b-23939d3eff20"
Jul 07 06:32:29.446433 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:29.446334    2563 generic.go:356] "Generic (PLEG): container finished" podID="0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c" containerID="96f3a3ec6e8807058cd80c80e14abe17d3fb5204b8b44bfc0ba56397e5bf1d47" exitCode=0
Jul 07 06:32:29.446433 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:29.446418    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b6cs5" event={"ID":"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c","Type":"ContainerDied","Data":"96f3a3ec6e8807058cd80c80e14abe17d3fb5204b8b44bfc0ba56397e5bf1d47"}
Jul 07 06:32:30.451131 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:30.451041    2563 generic.go:356] "Generic (PLEG): container finished" podID="0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c" containerID="75f3784de0002df30dd19f1aeae4eeb11011321f58d83d7c24d97a8421a1374d" exitCode=0
Jul 07 06:32:30.451131 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:30.451081    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b6cs5" event={"ID":"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c","Type":"ContainerDied","Data":"75f3784de0002df30dd19f1aeae4eeb11011321f58d83d7c24d97a8421a1374d"}
Jul 07 06:32:30.805382 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:30.805354    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee9fd62f-81cf-4544-8454-9fa9f53c0768-original-pull-secret\") pod \"global-pull-secret-syncer-57vh8\" (UID: \"ee9fd62f-81cf-4544-8454-9fa9f53c0768\") " pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:30.805531 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:30.805399    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c1e103c-f156-4b8a-8426-f18974595c51-metrics-certs\") pod \"network-metrics-daemon-mx8qb\" (UID: \"7c1e103c-f156-4b8a-8426-f18974595c51\") " pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:30.805531 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:30.805488    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:32:30.805531 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:30.805489    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 06:32:30.805680 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:30.805542    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c1e103c-f156-4b8a-8426-f18974595c51-metrics-certs podName:7c1e103c-f156-4b8a-8426-f18974595c51 nodeName:}" failed. No retries permitted until 2026-07-07 06:33:02.805528219 +0000 UTC m=+66.169724742 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c1e103c-f156-4b8a-8426-f18974595c51-metrics-certs") pod "network-metrics-daemon-mx8qb" (UID: "7c1e103c-f156-4b8a-8426-f18974595c51") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 06:32:30.805680 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:30.805556    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee9fd62f-81cf-4544-8454-9fa9f53c0768-original-pull-secret podName:ee9fd62f-81cf-4544-8454-9fa9f53c0768 nodeName:}" failed. No retries permitted until 2026-07-07 06:33:02.805549754 +0000 UTC m=+66.169746277 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ee9fd62f-81cf-4544-8454-9fa9f53c0768-original-pull-secret") pod "global-pull-secret-syncer-57vh8" (UID: "ee9fd62f-81cf-4544-8454-9fa9f53c0768") : object "kube-system"/"original-pull-secret" not registered
Jul 07 06:32:31.007223 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:31.007188    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47ljz\" (UniqueName: \"kubernetes.io/projected/ffb473f1-6af6-40d5-b15b-23939d3eff20-kube-api-access-47ljz\") pod \"network-check-target-jlprw\" (UID: \"ffb473f1-6af6-40d5-b15b-23939d3eff20\") " pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:31.007383 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:31.007342    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 06:32:31.007383 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:31.007364    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 06:32:31.007383 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:31.007374    2563 projected.go:196] Error preparing data for projected volume kube-api-access-47ljz for pod openshift-network-diagnostics/network-check-target-jlprw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:32:31.007484 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:31.007430    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffb473f1-6af6-40d5-b15b-23939d3eff20-kube-api-access-47ljz podName:ffb473f1-6af6-40d5-b15b-23939d3eff20 nodeName:}" failed. No retries permitted until 2026-07-07 06:33:03.007414835 +0000 UTC m=+66.371611362 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-47ljz" (UniqueName: "kubernetes.io/projected/ffb473f1-6af6-40d5-b15b-23939d3eff20-kube-api-access-47ljz") pod "network-check-target-jlprw" (UID: "ffb473f1-6af6-40d5-b15b-23939d3eff20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 06:32:31.165713 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:31.165620    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:31.165713 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:31.165673    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:31.165911 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:31.165620    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:31.165911 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:31.165763    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx8qb" podUID="7c1e103c-f156-4b8a-8426-f18974595c51"
Jul 07 06:32:31.165911 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:31.165835    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57vh8" podUID="ee9fd62f-81cf-4544-8454-9fa9f53c0768"
Jul 07 06:32:31.165911 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:31.165891    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlprw" podUID="ffb473f1-6af6-40d5-b15b-23939d3eff20"
Jul 07 06:32:33.165893 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:33.165861    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:33.166263 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:33.165861    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:33.166263 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:33.165963    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx8qb" podUID="7c1e103c-f156-4b8a-8426-f18974595c51"
Jul 07 06:32:33.166263 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:33.165861    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:33.166263 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:33.166048    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57vh8" podUID="ee9fd62f-81cf-4544-8454-9fa9f53c0768"
Jul 07 06:32:33.166263 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:33.166100    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlprw" podUID="ffb473f1-6af6-40d5-b15b-23939d3eff20"
Jul 07 06:32:35.166445 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:35.166409    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:35.166869 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:35.166418    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:35.166869 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:35.166528    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:35.166869 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:35.166526    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlprw" podUID="ffb473f1-6af6-40d5-b15b-23939d3eff20"
Jul 07 06:32:35.166869 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:35.166584    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57vh8" podUID="ee9fd62f-81cf-4544-8454-9fa9f53c0768"
Jul 07 06:32:35.166869 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:35.166705    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx8qb" podUID="7c1e103c-f156-4b8a-8426-f18974595c51"
Jul 07 06:32:35.464844 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:35.464810    2563 generic.go:356] "Generic (PLEG): container finished" podID="0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c" containerID="230c887b783e8ae41b0496672ae63fcc066cf0de39e5496f2c74d6c9b629615f" exitCode=0
Jul 07 06:32:35.464999 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:35.464888    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b6cs5" event={"ID":"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c","Type":"ContainerDied","Data":"230c887b783e8ae41b0496672ae63fcc066cf0de39e5496f2c74d6c9b629615f"}
Jul 07 06:32:36.469756 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:36.469713    2563 generic.go:356] "Generic (PLEG): container finished" podID="0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c" containerID="2359e1d03b1e1d182c9ace2e55445a074c470fb6fd32316e642f2e25a9d762fd" exitCode=0
Jul 07 06:32:36.470158 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:36.469781    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b6cs5" event={"ID":"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c","Type":"ContainerDied","Data":"2359e1d03b1e1d182c9ace2e55445a074c470fb6fd32316e642f2e25a9d762fd"}
Jul 07 06:32:37.166821 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:37.166659    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:37.166983 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:37.166868    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57vh8" podUID="ee9fd62f-81cf-4544-8454-9fa9f53c0768"
Jul 07 06:32:37.166983 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:37.166750    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:37.166983 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:37.166928    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx8qb" podUID="7c1e103c-f156-4b8a-8426-f18974595c51"
Jul 07 06:32:37.166983 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:37.166722    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:37.167202 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:37.166995    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlprw" podUID="ffb473f1-6af6-40d5-b15b-23939d3eff20"
Jul 07 06:32:37.474821 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:37.474745    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b6cs5" event={"ID":"0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c","Type":"ContainerStarted","Data":"83e60e98eca55cc14546264a387b4cb755a07d4d0e170f3b32dcfb4b8e3f6e53"}
Jul 07 06:32:37.498583 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:37.498543    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b6cs5" podStartSLOduration=8.935670473 podStartE2EDuration="40.498529084s" totalImagesPullingTime="31.562858611s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 06:31:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:32:37.497937717 +0000 UTC m=+40.862134262" watchObservedRunningTime="2026-07-07 06:32:37.498529084 +0000 UTC m=+40.862725628"
Jul 07 06:32:39.165813 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:39.165770    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:39.165813 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:39.165785    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:39.166327 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:39.165903    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:39.166327 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:39.165918    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx8qb" podUID="7c1e103c-f156-4b8a-8426-f18974595c51"
Jul 07 06:32:39.166327 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:39.165995    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlprw" podUID="ffb473f1-6af6-40d5-b15b-23939d3eff20"
Jul 07 06:32:39.166327 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:39.166149    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57vh8" podUID="ee9fd62f-81cf-4544-8454-9fa9f53c0768"
Jul 07 06:32:41.120059 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:41.120019    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-57vh8"]
Jul 07 06:32:41.120871 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:41.120153    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:41.120871 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:41.120279    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57vh8" podUID="ee9fd62f-81cf-4544-8454-9fa9f53c0768"
Jul 07 06:32:41.122791 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:41.122763    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jlprw"]
Jul 07 06:32:41.122945 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:41.122871    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:41.122993 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:41.122961    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlprw" podUID="ffb473f1-6af6-40d5-b15b-23939d3eff20"
Jul 07 06:32:41.123479 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:41.123455    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mx8qb"]
Jul 07 06:32:41.123590 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:41.123548    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:41.123689 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:41.123668    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx8qb" podUID="7c1e103c-f156-4b8a-8426-f18974595c51"
Jul 07 06:32:42.165575 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:42.165547    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:42.166057 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:42.165674    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx8qb" podUID="7c1e103c-f156-4b8a-8426-f18974595c51"
Jul 07 06:32:43.165877 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:43.165839    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:43.166397 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:43.165954    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57vh8" podUID="ee9fd62f-81cf-4544-8454-9fa9f53c0768"
Jul 07 06:32:43.166397 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:43.166018    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:43.166397 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:43.166151    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlprw" podUID="ffb473f1-6af6-40d5-b15b-23939d3eff20"
Jul 07 06:32:44.165578 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:44.165544    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:44.165779 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:44.165679    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx8qb" podUID="7c1e103c-f156-4b8a-8426-f18974595c51"
Jul 07 06:32:45.166330 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.166290    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:45.166727 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.166338    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:45.166727 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:45.166425    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57vh8" podUID="ee9fd62f-81cf-4544-8454-9fa9f53c0768"
Jul 07 06:32:45.166727 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:45.166535    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlprw" podUID="ffb473f1-6af6-40d5-b15b-23939d3eff20"
Jul 07 06:32:45.440919 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.440891    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-92.ec2.internal" event="NodeReady"
Jul 07 06:32:45.441085 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.441021    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 07 06:32:45.508897 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.508861    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gvkjl"]
Jul 07 06:32:45.518234 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.518209    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gvkjl"
Jul 07 06:32:45.520652 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.520612    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 07 06:32:45.521343 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.521322    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 07 06:32:45.521472 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.521457    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lfbmp\""
Jul 07 06:32:45.524971 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.524951    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gvkjl"]
Jul 07 06:32:45.530570 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.530546    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vhq4w"]
Jul 07 06:32:45.540070 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.540047    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vhq4w"
Jul 07 06:32:45.542138 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.542119    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 07 06:32:45.542264 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.542124    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 07 06:32:45.542938 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.542920    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 07 06:32:45.543730 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.543705    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 07 06:32:45.543827 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.543730    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-68w6b\""
Jul 07 06:32:45.544491 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.544471    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vhq4w"]
Jul 07 06:32:45.586135 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.585942    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qmc4r"]
Jul 07 06:32:45.603627 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.603583    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qmc4r"]
Jul 07 06:32:45.603822 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.603698    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qmc4r"
Jul 07 06:32:45.606057 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.606035    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 07 06:32:45.606195 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.606081    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 07 06:32:45.606441 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.606426    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bdntf\""
Jul 07 06:32:45.606574 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.606556    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 07 06:32:45.612028 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.612009    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e5f0669f-f5b0-4d36-b791-681380b94eb6-crio-socket\") pod \"insights-runtime-extractor-vhq4w\" (UID: \"e5f0669f-f5b0-4d36-b791-681380b94eb6\") " pod="openshift-insights/insights-runtime-extractor-vhq4w"
Jul 07 06:32:45.612141 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.612038    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5f0669f-f5b0-4d36-b791-681380b94eb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhq4w\" (UID: \"e5f0669f-f5b0-4d36-b791-681380b94eb6\") " pod="openshift-insights/insights-runtime-extractor-vhq4w"
Jul 07 06:32:45.612141 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.612056    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlxg9\" (UniqueName: \"kubernetes.io/projected/e5f0669f-f5b0-4d36-b791-681380b94eb6-kube-api-access-mlxg9\") pod \"insights-runtime-extractor-vhq4w\" (UID: \"e5f0669f-f5b0-4d36-b791-681380b94eb6\") " pod="openshift-insights/insights-runtime-extractor-vhq4w"
Jul 07 06:32:45.612141 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.612120    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c196cdf7-b0a5-4fa4-8f02-9176334b6340-tmp-dir\") pod \"dns-default-gvkjl\" (UID: \"c196cdf7-b0a5-4fa4-8f02-9176334b6340\") " pod="openshift-dns/dns-default-gvkjl"
Jul 07 06:32:45.612285 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.612149    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c196cdf7-b0a5-4fa4-8f02-9176334b6340-config-volume\") pod \"dns-default-gvkjl\" (UID: \"c196cdf7-b0a5-4fa4-8f02-9176334b6340\") " pod="openshift-dns/dns-default-gvkjl"
Jul 07 06:32:45.612285 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.612193    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c196cdf7-b0a5-4fa4-8f02-9176334b6340-metrics-tls\") pod \"dns-default-gvkjl\" (UID: \"c196cdf7-b0a5-4fa4-8f02-9176334b6340\") " pod="openshift-dns/dns-default-gvkjl"
Jul 07 06:32:45.612285 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.612275    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5f0669f-f5b0-4d36-b791-681380b94eb6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhq4w\" (UID: \"e5f0669f-f5b0-4d36-b791-681380b94eb6\") " pod="openshift-insights/insights-runtime-extractor-vhq4w"
Jul 07 06:32:45.612382 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.612322    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq7t4\" (UniqueName: \"kubernetes.io/projected/c196cdf7-b0a5-4fa4-8f02-9176334b6340-kube-api-access-tq7t4\") pod \"dns-default-gvkjl\" (UID: \"c196cdf7-b0a5-4fa4-8f02-9176334b6340\") " pod="openshift-dns/dns-default-gvkjl"
Jul 07 06:32:45.612382 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.612356    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5f0669f-f5b0-4d36-b791-681380b94eb6-data-volume\") pod \"insights-runtime-extractor-vhq4w\" (UID: \"e5f0669f-f5b0-4d36-b791-681380b94eb6\") " pod="openshift-insights/insights-runtime-extractor-vhq4w"
Jul 07 06:32:45.697796 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.697712    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f45b5f89f-t2pnz"]
Jul 07 06:32:45.710020 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.709991    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f45b5f89f-t2pnz"
Jul 07 06:32:45.712697 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.712669    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxbhz\" (UniqueName: \"kubernetes.io/projected/0cc3267d-e426-4fa1-88c9-9fbdcda4e8ee-kube-api-access-zxbhz\") pod \"ingress-canary-qmc4r\" (UID: \"0cc3267d-e426-4fa1-88c9-9fbdcda4e8ee\") " pod="openshift-ingress-canary/ingress-canary-qmc4r"
Jul 07 06:32:45.712924 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.712798    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 07 06:32:45.713023 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.712848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq7t4\" (UniqueName: \"kubernetes.io/projected/c196cdf7-b0a5-4fa4-8f02-9176334b6340-kube-api-access-tq7t4\") pod \"dns-default-gvkjl\" (UID: \"c196cdf7-b0a5-4fa4-8f02-9176334b6340\") " pod="openshift-dns/dns-default-gvkjl"
Jul 07 06:32:45.713088 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.713021    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5f0669f-f5b0-4d36-b791-681380b94eb6-data-volume\") pod \"insights-runtime-extractor-vhq4w\" (UID: \"e5f0669f-f5b0-4d36-b791-681380b94eb6\") " pod="openshift-insights/insights-runtime-extractor-vhq4w"
Jul 07 06:32:45.713088 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.713030    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 07 06:32:45.713088 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.713039    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 07 06:32:45.713088 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.713056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e5f0669f-f5b0-4d36-b791-681380b94eb6-crio-socket\") pod \"insights-runtime-extractor-vhq4w\" (UID: \"e5f0669f-f5b0-4d36-b791-681380b94eb6\") " pod="openshift-insights/insights-runtime-extractor-vhq4w"
Jul 07 06:32:45.713273 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.713090    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5f0669f-f5b0-4d36-b791-681380b94eb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhq4w\" (UID: \"e5f0669f-f5b0-4d36-b791-681380b94eb6\") " pod="openshift-insights/insights-runtime-extractor-vhq4w"
Jul 07 06:32:45.713273 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.713113    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlxg9\" (UniqueName: \"kubernetes.io/projected/e5f0669f-f5b0-4d36-b791-681380b94eb6-kube-api-access-mlxg9\") pod \"insights-runtime-extractor-vhq4w\" (UID: \"e5f0669f-f5b0-4d36-b791-681380b94eb6\") " pod="openshift-insights/insights-runtime-extractor-vhq4w"
Jul 07 06:32:45.713273 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.713161    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c196cdf7-b0a5-4fa4-8f02-9176334b6340-tmp-dir\") pod \"dns-default-gvkjl\" (UID: \"c196cdf7-b0a5-4fa4-8f02-9176334b6340\") " pod="openshift-dns/dns-default-gvkjl"
Jul 07 06:32:45.713273 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.713185    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cc3267d-e426-4fa1-88c9-9fbdcda4e8ee-cert\") pod \"ingress-canary-qmc4r\" (UID: \"0cc3267d-e426-4fa1-88c9-9fbdcda4e8ee\") " pod="openshift-ingress-canary/ingress-canary-qmc4r"
Jul 07 06:32:45.713273 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.713211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c196cdf7-b0a5-4fa4-8f02-9176334b6340-config-volume\") pod \"dns-default-gvkjl\" (UID: \"c196cdf7-b0a5-4fa4-8f02-9176334b6340\") " pod="openshift-dns/dns-default-gvkjl"
Jul 07 06:32:45.713273 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.713225    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c196cdf7-b0a5-4fa4-8f02-9176334b6340-metrics-tls\") pod \"dns-default-gvkjl\" (UID: \"c196cdf7-b0a5-4fa4-8f02-9176334b6340\") " pod="openshift-dns/dns-default-gvkjl"
Jul 07 06:32:45.713273 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.712948    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 07 06:32:45.713273 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.713249    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5f0669f-f5b0-4d36-b791-681380b94eb6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhq4w\" (UID: \"e5f0669f-f5b0-4d36-b791-681380b94eb6\") " pod="openshift-insights/insights-runtime-extractor-vhq4w"
Jul 07 06:32:45.713669 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.713327    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e5f0669f-f5b0-4d36-b791-681380b94eb6-crio-socket\") pod \"insights-runtime-extractor-vhq4w\" (UID: \"e5f0669f-f5b0-4d36-b791-681380b94eb6\") " pod="openshift-insights/insights-runtime-extractor-vhq4w"
Jul 07 06:32:45.713669 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.713445    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5f0669f-f5b0-4d36-b791-681380b94eb6-data-volume\") pod \"insights-runtime-extractor-vhq4w\" (UID: \"e5f0669f-f5b0-4d36-b791-681380b94eb6\") " pod="openshift-insights/insights-runtime-extractor-vhq4w"
Jul 07 06:32:45.713957 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.713931    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 07 06:32:45.714058 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.713966    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c196cdf7-b0a5-4fa4-8f02-9176334b6340-config-volume\") pod \"dns-default-gvkjl\" (UID: \"c196cdf7-b0a5-4fa4-8f02-9176334b6340\") " pod="openshift-dns/dns-default-gvkjl"
Jul 07 06:32:45.714058 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.713934    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5f0669f-f5b0-4d36-b791-681380b94eb6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhq4w\" (UID: \"e5f0669f-f5b0-4d36-b791-681380b94eb6\") " pod="openshift-insights/insights-runtime-extractor-vhq4w"
Jul 07 06:32:45.714218 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.714198    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 07 06:32:45.714296 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.714223    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 07 06:32:45.714296 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.714280    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c196cdf7-b0a5-4fa4-8f02-9176334b6340-tmp-dir\") pod \"dns-default-gvkjl\" (UID: \"c196cdf7-b0a5-4fa4-8f02-9176334b6340\") " pod="openshift-dns/dns-default-gvkjl"
Jul 07 06:32:45.714485 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.714466    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mld6g\""
Jul 07 06:32:45.715339 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.715292    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f45b5f89f-t2pnz"]
Jul 07 06:32:45.718774 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.718752    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5f0669f-f5b0-4d36-b791-681380b94eb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhq4w\" (UID: \"e5f0669f-f5b0-4d36-b791-681380b94eb6\") " pod="openshift-insights/insights-runtime-extractor-vhq4w"
Jul 07 06:32:45.718860 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.718760    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c196cdf7-b0a5-4fa4-8f02-9176334b6340-metrics-tls\") pod \"dns-default-gvkjl\" (UID: \"c196cdf7-b0a5-4fa4-8f02-9176334b6340\") " pod="openshift-dns/dns-default-gvkjl"
Jul 07 06:32:45.722126 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.722104    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlxg9\" (UniqueName: \"kubernetes.io/projected/e5f0669f-f5b0-4d36-b791-681380b94eb6-kube-api-access-mlxg9\") pod \"insights-runtime-extractor-vhq4w\" (UID: \"e5f0669f-f5b0-4d36-b791-681380b94eb6\") " pod="openshift-insights/insights-runtime-extractor-vhq4w"
Jul 07 06:32:45.732245 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.732218    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq7t4\" (UniqueName: \"kubernetes.io/projected/c196cdf7-b0a5-4fa4-8f02-9176334b6340-kube-api-access-tq7t4\") pod \"dns-default-gvkjl\" (UID: \"c196cdf7-b0a5-4fa4-8f02-9176334b6340\") " pod="openshift-dns/dns-default-gvkjl"
Jul 07 06:32:45.814137 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.814095    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6427e28-9acc-4588-ac81-a9c40bdee78e-oauth-serving-cert\") pod \"console-7f45b5f89f-t2pnz\" (UID: \"b6427e28-9acc-4588-ac81-a9c40bdee78e\") " pod="openshift-console/console-7f45b5f89f-t2pnz"
Jul 07 06:32:45.814323 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.814156    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cc3267d-e426-4fa1-88c9-9fbdcda4e8ee-cert\") pod \"ingress-canary-qmc4r\" (UID: \"0cc3267d-e426-4fa1-88c9-9fbdcda4e8ee\") " pod="openshift-ingress-canary/ingress-canary-qmc4r"
Jul 07 06:32:45.814323 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.814178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6427e28-9acc-4588-ac81-a9c40bdee78e-service-ca\") pod \"console-7f45b5f89f-t2pnz\" (UID: \"b6427e28-9acc-4588-ac81-a9c40bdee78e\") " pod="openshift-console/console-7f45b5f89f-t2pnz"
Jul 07 06:32:45.814323 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.814195    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6427e28-9acc-4588-ac81-a9c40bdee78e-console-serving-cert\") pod \"console-7f45b5f89f-t2pnz\" (UID: \"b6427e28-9acc-4588-ac81-a9c40bdee78e\") " pod="openshift-console/console-7f45b5f89f-t2pnz"
Jul 07 06:32:45.814323 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.814214    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6427e28-9acc-4588-ac81-a9c40bdee78e-console-config\") pod \"console-7f45b5f89f-t2pnz\" (UID: \"b6427e28-9acc-4588-ac81-a9c40bdee78e\") " pod="openshift-console/console-7f45b5f89f-t2pnz"
Jul 07 06:32:45.814323 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.814235    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxbhz\" (UniqueName: \"kubernetes.io/projected/0cc3267d-e426-4fa1-88c9-9fbdcda4e8ee-kube-api-access-zxbhz\") pod \"ingress-canary-qmc4r\" (UID: \"0cc3267d-e426-4fa1-88c9-9fbdcda4e8ee\") " pod="openshift-ingress-canary/ingress-canary-qmc4r"
Jul 07 06:32:45.814323 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.814264    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qtpk\" (UniqueName: \"kubernetes.io/projected/b6427e28-9acc-4588-ac81-a9c40bdee78e-kube-api-access-9qtpk\") pod \"console-7f45b5f89f-t2pnz\" (UID: \"b6427e28-9acc-4588-ac81-a9c40bdee78e\") " pod="openshift-console/console-7f45b5f89f-t2pnz"
Jul 07 06:32:45.814559 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.814326    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6427e28-9acc-4588-ac81-a9c40bdee78e-console-oauth-config\") pod \"console-7f45b5f89f-t2pnz\" (UID: \"b6427e28-9acc-4588-ac81-a9c40bdee78e\") " pod="openshift-console/console-7f45b5f89f-t2pnz"
Jul 07 06:32:45.816685 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.816667    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cc3267d-e426-4fa1-88c9-9fbdcda4e8ee-cert\") pod \"ingress-canary-qmc4r\" (UID: \"0cc3267d-e426-4fa1-88c9-9fbdcda4e8ee\") " pod="openshift-ingress-canary/ingress-canary-qmc4r"
Jul 07 06:32:45.821738 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.821711    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxbhz\" (UniqueName: \"kubernetes.io/projected/0cc3267d-e426-4fa1-88c9-9fbdcda4e8ee-kube-api-access-zxbhz\") pod \"ingress-canary-qmc4r\" (UID: \"0cc3267d-e426-4fa1-88c9-9fbdcda4e8ee\") " pod="openshift-ingress-canary/ingress-canary-qmc4r"
Jul 07 06:32:45.829540 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.829510    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gvkjl"
Jul 07 06:32:45.850341 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.850310    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vhq4w"
Jul 07 06:32:45.913574 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.913462    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qmc4r"
Jul 07 06:32:45.915435 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.914885    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6427e28-9acc-4588-ac81-a9c40bdee78e-service-ca\") pod \"console-7f45b5f89f-t2pnz\" (UID: \"b6427e28-9acc-4588-ac81-a9c40bdee78e\") " pod="openshift-console/console-7f45b5f89f-t2pnz"
Jul 07 06:32:45.915435 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.914928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6427e28-9acc-4588-ac81-a9c40bdee78e-console-serving-cert\") pod \"console-7f45b5f89f-t2pnz\" (UID: \"b6427e28-9acc-4588-ac81-a9c40bdee78e\") " pod="openshift-console/console-7f45b5f89f-t2pnz"
Jul 07 06:32:45.915435 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.914954    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6427e28-9acc-4588-ac81-a9c40bdee78e-console-config\") pod \"console-7f45b5f89f-t2pnz\" (UID: \"b6427e28-9acc-4588-ac81-a9c40bdee78e\") " pod="openshift-console/console-7f45b5f89f-t2pnz"
Jul 07 06:32:45.915435 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.915006    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qtpk\" (UniqueName: \"kubernetes.io/projected/b6427e28-9acc-4588-ac81-a9c40bdee78e-kube-api-access-9qtpk\") pod \"console-7f45b5f89f-t2pnz\" (UID: \"b6427e28-9acc-4588-ac81-a9c40bdee78e\") " pod="openshift-console/console-7f45b5f89f-t2pnz"
Jul 07 06:32:45.915435 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.915045    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6427e28-9acc-4588-ac81-a9c40bdee78e-console-oauth-config\") pod \"console-7f45b5f89f-t2pnz\" (UID: \"b6427e28-9acc-4588-ac81-a9c40bdee78e\") " pod="openshift-console/console-7f45b5f89f-t2pnz"
Jul 07 06:32:45.915435 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.915090    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6427e28-9acc-4588-ac81-a9c40bdee78e-oauth-serving-cert\") pod \"console-7f45b5f89f-t2pnz\" (UID: \"b6427e28-9acc-4588-ac81-a9c40bdee78e\") " pod="openshift-console/console-7f45b5f89f-t2pnz"
Jul 07 06:32:45.915862 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.915540    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6427e28-9acc-4588-ac81-a9c40bdee78e-service-ca\") pod \"console-7f45b5f89f-t2pnz\" (UID: \"b6427e28-9acc-4588-ac81-a9c40bdee78e\") " pod="openshift-console/console-7f45b5f89f-t2pnz"
Jul 07 06:32:45.916449 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.916189    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6427e28-9acc-4588-ac81-a9c40bdee78e-oauth-serving-cert\") pod \"console-7f45b5f89f-t2pnz\" (UID: \"b6427e28-9acc-4588-ac81-a9c40bdee78e\") " pod="openshift-console/console-7f45b5f89f-t2pnz"
Jul 07 06:32:45.916825 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.916562    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6427e28-9acc-4588-ac81-a9c40bdee78e-console-config\") pod \"console-7f45b5f89f-t2pnz\" (UID: \"b6427e28-9acc-4588-ac81-a9c40bdee78e\") " pod="openshift-console/console-7f45b5f89f-t2pnz"
Jul 07 06:32:45.918975 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.918947    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6427e28-9acc-4588-ac81-a9c40bdee78e-console-serving-cert\") pod \"console-7f45b5f89f-t2pnz\" (UID: \"b6427e28-9acc-4588-ac81-a9c40bdee78e\") " pod="openshift-console/console-7f45b5f89f-t2pnz"
Jul 07 06:32:45.919561 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.919514    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6427e28-9acc-4588-ac81-a9c40bdee78e-console-oauth-config\") pod \"console-7f45b5f89f-t2pnz\" (UID: \"b6427e28-9acc-4588-ac81-a9c40bdee78e\") " pod="openshift-console/console-7f45b5f89f-t2pnz"
Jul 07 06:32:45.931170 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:45.931139    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qtpk\" (UniqueName: \"kubernetes.io/projected/b6427e28-9acc-4588-ac81-a9c40bdee78e-kube-api-access-9qtpk\") pod \"console-7f45b5f89f-t2pnz\" (UID: \"b6427e28-9acc-4588-ac81-a9c40bdee78e\") " pod="openshift-console/console-7f45b5f89f-t2pnz"
Jul 07 06:32:46.004244 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:46.004094    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gvkjl"]
Jul 07 06:32:46.006332 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:46.006296    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vhq4w"]
Jul 07 06:32:46.007569 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:32:46.007545    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc196cdf7_b0a5_4fa4_8f02_9176334b6340.slice/crio-8e6bf9495064b151a8a4744f07bc944c6ca7933951f7b9c50623162cc2b335fd WatchSource:0}: Error finding container 8e6bf9495064b151a8a4744f07bc944c6ca7933951f7b9c50623162cc2b335fd: Status 404 returned error can't find the container with id 8e6bf9495064b151a8a4744f07bc944c6ca7933951f7b9c50623162cc2b335fd
Jul 07 06:32:46.010700 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:32:46.010624    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f0669f_f5b0_4d36_b791_681380b94eb6.slice/crio-1e7fcef363b145d294d8bdec2bb99f6caf6caa4d39873d9fc2b8a232011b6d94 WatchSource:0}: Error finding container 1e7fcef363b145d294d8bdec2bb99f6caf6caa4d39873d9fc2b8a232011b6d94: Status 404 returned error can't find the container with id 1e7fcef363b145d294d8bdec2bb99f6caf6caa4d39873d9fc2b8a232011b6d94
Jul 07 06:32:46.030014 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:46.029969    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f45b5f89f-t2pnz"
Jul 07 06:32:46.071118 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:46.071081    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qmc4r"]
Jul 07 06:32:46.075436 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:32:46.075408    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc3267d_e426_4fa1_88c9_9fbdcda4e8ee.slice/crio-b6f4193a61e2408717ce929605c5610737fafd795078c013fff28a63f9f30e56 WatchSource:0}: Error finding container b6f4193a61e2408717ce929605c5610737fafd795078c013fff28a63f9f30e56: Status 404 returned error can't find the container with id b6f4193a61e2408717ce929605c5610737fafd795078c013fff28a63f9f30e56
Jul 07 06:32:46.164813 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:46.164783    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f45b5f89f-t2pnz"]
Jul 07 06:32:46.166113 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:46.166096    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:32:46.167760 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:32:46.167738    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6427e28_9acc_4588_ac81_a9c40bdee78e.slice/crio-3ab9a1df96835eb00a45a7f8c866e8e4de8d72587302e7c267b68203bf78a962 WatchSource:0}: Error finding container 3ab9a1df96835eb00a45a7f8c866e8e4de8d72587302e7c267b68203bf78a962: Status 404 returned error can't find the container with id 3ab9a1df96835eb00a45a7f8c866e8e4de8d72587302e7c267b68203bf78a962
Jul 07 06:32:46.168242 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:46.168218    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 06:32:46.168318 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:46.168270    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5wsxr\""
Jul 07 06:32:46.495403 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:46.495365    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vhq4w" event={"ID":"e5f0669f-f5b0-4d36-b791-681380b94eb6","Type":"ContainerStarted","Data":"1e7fcef363b145d294d8bdec2bb99f6caf6caa4d39873d9fc2b8a232011b6d94"}
Jul 07 06:32:46.496323 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:46.496305    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gvkjl" event={"ID":"c196cdf7-b0a5-4fa4-8f02-9176334b6340","Type":"ContainerStarted","Data":"8e6bf9495064b151a8a4744f07bc944c6ca7933951f7b9c50623162cc2b335fd"}
Jul 07 06:32:46.497319 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:46.497297    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qmc4r" event={"ID":"0cc3267d-e426-4fa1-88c9-9fbdcda4e8ee","Type":"ContainerStarted","Data":"b6f4193a61e2408717ce929605c5610737fafd795078c013fff28a63f9f30e56"}
Jul 07 06:32:46.498353 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:46.498337    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f45b5f89f-t2pnz" event={"ID":"b6427e28-9acc-4588-ac81-a9c40bdee78e","Type":"ContainerStarted","Data":"3ab9a1df96835eb00a45a7f8c866e8e4de8d72587302e7c267b68203bf78a962"}
Jul 07 06:32:47.172009 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:47.170485    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:32:47.172009 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:47.170923    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:32:47.177149 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:47.175433    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 06:32:47.177149 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:47.175731    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 06:32:47.177149 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:47.176333    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 06:32:47.183515 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:47.177432    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dsscv\""
Jul 07 06:32:51.713008 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:51.712777    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vwcc9"]
Jul 07 06:32:51.727065 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:51.727017    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vwcc9"]
Jul 07 06:32:51.727265 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:51.727153    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vwcc9"
Jul 07 06:32:51.729165 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:51.729135    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 07 06:32:51.729364 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:51.729166    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-cwtsd\""
Jul 07 06:32:51.765238 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:51.765202    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7f71831a-ee04-48df-a4f3-9636ab2da704-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-vwcc9\" (UID: \"7f71831a-ee04-48df-a4f3-9636ab2da704\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vwcc9"
Jul 07 06:32:51.865735 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:51.865702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7f71831a-ee04-48df-a4f3-9636ab2da704-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-vwcc9\" (UID: \"7f71831a-ee04-48df-a4f3-9636ab2da704\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vwcc9"
Jul 07 06:32:51.865904 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:51.865840    2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 07 06:32:51.865965 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:51.865907    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f71831a-ee04-48df-a4f3-9636ab2da704-tls-certificates podName:7f71831a-ee04-48df-a4f3-9636ab2da704 nodeName:}" failed. No retries permitted until 2026-07-07 06:32:52.365892387 +0000 UTC m=+55.730088910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/7f71831a-ee04-48df-a4f3-9636ab2da704-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-vwcc9" (UID: "7f71831a-ee04-48df-a4f3-9636ab2da704") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 07 06:32:52.369379 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:52.369348    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7f71831a-ee04-48df-a4f3-9636ab2da704-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-vwcc9\" (UID: \"7f71831a-ee04-48df-a4f3-9636ab2da704\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vwcc9"
Jul 07 06:32:52.369571 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:52.369503    2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 07 06:32:52.369571 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:32:52.369561    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f71831a-ee04-48df-a4f3-9636ab2da704-tls-certificates podName:7f71831a-ee04-48df-a4f3-9636ab2da704 nodeName:}" failed. No retries permitted until 2026-07-07 06:32:53.369546892 +0000 UTC m=+56.733743415 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/7f71831a-ee04-48df-a4f3-9636ab2da704-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-vwcc9" (UID: "7f71831a-ee04-48df-a4f3-9636ab2da704") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 07 06:32:53.378483 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:53.378440    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7f71831a-ee04-48df-a4f3-9636ab2da704-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-vwcc9\" (UID: \"7f71831a-ee04-48df-a4f3-9636ab2da704\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vwcc9"
Jul 07 06:32:53.382717 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:53.382681    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7f71831a-ee04-48df-a4f3-9636ab2da704-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-vwcc9\" (UID: \"7f71831a-ee04-48df-a4f3-9636ab2da704\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vwcc9"
Jul 07 06:32:53.539363 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:53.539134    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vwcc9"
Jul 07 06:32:53.750616 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:53.750469    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vwcc9"]
Jul 07 06:32:53.753783 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:32:53.753745    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f71831a_ee04_48df_a4f3_9636ab2da704.slice/crio-282bc67f459984253d71ee5898fffdf67cecb8d09a535d7775bb8911cc0cd82e WatchSource:0}: Error finding container 282bc67f459984253d71ee5898fffdf67cecb8d09a535d7775bb8911cc0cd82e: Status 404 returned error can't find the container with id 282bc67f459984253d71ee5898fffdf67cecb8d09a535d7775bb8911cc0cd82e
Jul 07 06:32:54.482220 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.482187    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5886655f8b-qm2fn"]
Jul 07 06:32:54.484017 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.484001    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5886655f8b-qm2fn"
Jul 07 06:32:54.491016 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.490966    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 07 06:32:54.498696 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.498619    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5886655f8b-qm2fn"]
Jul 07 06:32:54.527183 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.526714    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f45b5f89f-t2pnz" event={"ID":"b6427e28-9acc-4588-ac81-a9c40bdee78e","Type":"ContainerStarted","Data":"8dc5be401f6efb45cf79869183277374510ed8c840115dbec5fec090acb3bf32"}
Jul 07 06:32:54.528922 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.528891    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vhq4w" event={"ID":"e5f0669f-f5b0-4d36-b791-681380b94eb6","Type":"ContainerStarted","Data":"50ca23efb5d4940edf92ff7250f61f2ec95262f3e69110d083028d6d31be52af"}
Jul 07 06:32:54.529089 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.528955    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vhq4w" event={"ID":"e5f0669f-f5b0-4d36-b791-681380b94eb6","Type":"ContainerStarted","Data":"172fe6e548b0498ed7ed44597d3b8f6cda97ad4d85cba0231d9c2da9d7247289"}
Jul 07 06:32:54.530996 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.530920    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gvkjl" event={"ID":"c196cdf7-b0a5-4fa4-8f02-9176334b6340","Type":"ContainerStarted","Data":"c0b8c6667731bf26902e55c29174eb02c13fca3aeb92af4fb54cb254c8da1e89"}
Jul 07 06:32:54.530996 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.530977    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gvkjl" event={"ID":"c196cdf7-b0a5-4fa4-8f02-9176334b6340","Type":"ContainerStarted","Data":"22ae3c8a1c3bfe0cea16290c088c9185ab540ca73eac3ee479fa29bc88759e98"}
Jul 07 06:32:54.531190 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.531070    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gvkjl"
Jul 07 06:32:54.532436 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.532409    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vwcc9" event={"ID":"7f71831a-ee04-48df-a4f3-9636ab2da704","Type":"ContainerStarted","Data":"282bc67f459984253d71ee5898fffdf67cecb8d09a535d7775bb8911cc0cd82e"}
Jul 07 06:32:54.534147 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.534122    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qmc4r" event={"ID":"0cc3267d-e426-4fa1-88c9-9fbdcda4e8ee","Type":"ContainerStarted","Data":"5705b75b2cbe236f8679f05d030c40f3d00ca3be78406c5d601ae723902a0369"}
Jul 07 06:32:54.544325 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.544240    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f45b5f89f-t2pnz" podStartSLOduration=2.19603633 podStartE2EDuration="9.544218672s" totalImagesPullingTime="7.348182342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:32:54.543501716 +0000 UTC m=+57.907698263" watchObservedRunningTime="2026-07-07 06:32:54.544218672 +0000 UTC m=+57.908415217"
Jul 07 06:32:54.559437 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.559375    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qmc4r" podStartSLOduration=2.139267815 podStartE2EDuration="9.559355701s" totalImagesPullingTime="7.420087886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:32:54.558385296 +0000 UTC m=+57.922581858" watchObservedRunningTime="2026-07-07 06:32:54.559355701 +0000 UTC m=+57.923552240"
Jul 07 06:32:54.578810 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.578164    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gvkjl" podStartSLOduration=2.09586745 podStartE2EDuration="9.578140111s" totalImagesPullingTime="7.482272661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:32:54.577620397 +0000 UTC m=+57.941816962" watchObservedRunningTime="2026-07-07 06:32:54.578140111 +0000 UTC m=+57.942336639"
Jul 07 06:32:54.589121 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.589079    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/feddeb52-1647-49c2-98ab-7cfdfade2460-console-serving-cert\") pod \"console-5886655f8b-qm2fn\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") " pod="openshift-console/console-5886655f8b-qm2fn"
Jul 07 06:32:54.589306 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.589130    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/feddeb52-1647-49c2-98ab-7cfdfade2460-console-oauth-config\") pod \"console-5886655f8b-qm2fn\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") " pod="openshift-console/console-5886655f8b-qm2fn"
Jul 07 06:32:54.589306 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.589147    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/feddeb52-1647-49c2-98ab-7cfdfade2460-console-config\") pod \"console-5886655f8b-qm2fn\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") " pod="openshift-console/console-5886655f8b-qm2fn"
Jul 07 06:32:54.589306 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.589166    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feddeb52-1647-49c2-98ab-7cfdfade2460-trusted-ca-bundle\") pod \"console-5886655f8b-qm2fn\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") " pod="openshift-console/console-5886655f8b-qm2fn"
Jul 07 06:32:54.589306 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.589190    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/feddeb52-1647-49c2-98ab-7cfdfade2460-service-ca\") pod \"console-5886655f8b-qm2fn\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") " pod="openshift-console/console-5886655f8b-qm2fn"
Jul 07 06:32:54.589306 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.589250    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/feddeb52-1647-49c2-98ab-7cfdfade2460-oauth-serving-cert\") pod \"console-5886655f8b-qm2fn\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") " pod="openshift-console/console-5886655f8b-qm2fn"
Jul 07 06:32:54.589306 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.589271    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg59f\" (UniqueName: \"kubernetes.io/projected/feddeb52-1647-49c2-98ab-7cfdfade2460-kube-api-access-fg59f\") pod \"console-5886655f8b-qm2fn\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") " pod="openshift-console/console-5886655f8b-qm2fn"
Jul 07 06:32:54.690276 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.690235    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/feddeb52-1647-49c2-98ab-7cfdfade2460-console-serving-cert\") pod \"console-5886655f8b-qm2fn\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") " pod="openshift-console/console-5886655f8b-qm2fn"
Jul 07 06:32:54.690276 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.690283    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/feddeb52-1647-49c2-98ab-7cfdfade2460-console-oauth-config\") pod \"console-5886655f8b-qm2fn\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") " pod="openshift-console/console-5886655f8b-qm2fn"
Jul 07 06:32:54.690524 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.690307    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/feddeb52-1647-49c2-98ab-7cfdfade2460-console-config\") pod \"console-5886655f8b-qm2fn\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") " pod="openshift-console/console-5886655f8b-qm2fn"
Jul 07 06:32:54.690524 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.690336    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feddeb52-1647-49c2-98ab-7cfdfade2460-trusted-ca-bundle\") pod \"console-5886655f8b-qm2fn\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") " pod="openshift-console/console-5886655f8b-qm2fn"
Jul 07 06:32:54.690524 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.690366    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/feddeb52-1647-49c2-98ab-7cfdfade2460-service-ca\") pod \"console-5886655f8b-qm2fn\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") " pod="openshift-console/console-5886655f8b-qm2fn"
Jul 07 06:32:54.690524 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.690410    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/feddeb52-1647-49c2-98ab-7cfdfade2460-oauth-serving-cert\") pod \"console-5886655f8b-qm2fn\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") " pod="openshift-console/console-5886655f8b-qm2fn"
Jul 07 06:32:54.690524 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.690425    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg59f\" (UniqueName: \"kubernetes.io/projected/feddeb52-1647-49c2-98ab-7cfdfade2460-kube-api-access-fg59f\") pod \"console-5886655f8b-qm2fn\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") " pod="openshift-console/console-5886655f8b-qm2fn"
Jul 07 06:32:54.692672 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.691707    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/feddeb52-1647-49c2-98ab-7cfdfade2460-service-ca\") pod \"console-5886655f8b-qm2fn\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") " pod="openshift-console/console-5886655f8b-qm2fn"
Jul 07 06:32:54.692672 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.692039    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/feddeb52-1647-49c2-98ab-7cfdfade2460-oauth-serving-cert\") pod \"console-5886655f8b-qm2fn\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") " pod="openshift-console/console-5886655f8b-qm2fn"
Jul 07 06:32:54.692672 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.692548    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/feddeb52-1647-49c2-98ab-7cfdfade2460-console-config\") pod \"console-5886655f8b-qm2fn\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") " pod="openshift-console/console-5886655f8b-qm2fn"
Jul 07 06:32:54.692930 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.692686    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feddeb52-1647-49c2-98ab-7cfdfade2460-trusted-ca-bundle\") pod \"console-5886655f8b-qm2fn\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") " pod="openshift-console/console-5886655f8b-qm2fn"
Jul 07 06:32:54.693999 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.693966    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/feddeb52-1647-49c2-98ab-7cfdfade2460-console-oauth-config\") pod \"console-5886655f8b-qm2fn\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") " pod="openshift-console/console-5886655f8b-qm2fn"
Jul 07 06:32:54.693999 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.693993    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/feddeb52-1647-49c2-98ab-7cfdfade2460-console-serving-cert\") pod \"console-5886655f8b-qm2fn\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") " pod="openshift-console/console-5886655f8b-qm2fn"
Jul 07 06:32:54.699885 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.699859    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg59f\" (UniqueName: \"kubernetes.io/projected/feddeb52-1647-49c2-98ab-7cfdfade2460-kube-api-access-fg59f\") pod \"console-5886655f8b-qm2fn\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") " pod="openshift-console/console-5886655f8b-qm2fn"
Jul 07 06:32:54.795735 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:54.795590    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5886655f8b-qm2fn"
Jul 07 06:32:55.231901 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:55.231864    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5886655f8b-qm2fn"]
Jul 07 06:32:55.235103 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:32:55.235072    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeddeb52_1647_49c2_98ab_7cfdfade2460.slice/crio-d018dbf9919e25a4f590db314a6463cadbf755eadb42fff694870e829149a9f6 WatchSource:0}: Error finding container d018dbf9919e25a4f590db314a6463cadbf755eadb42fff694870e829149a9f6: Status 404 returned error can't find the container with id d018dbf9919e25a4f590db314a6463cadbf755eadb42fff694870e829149a9f6
Jul 07 06:32:55.540203 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:55.540105    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vwcc9" event={"ID":"7f71831a-ee04-48df-a4f3-9636ab2da704","Type":"ContainerStarted","Data":"3c62cdf51afd1c8bad276a8e4e1dfb5acd180cfcc2088c2d3d54c02ecad5e201"}
Jul 07 06:32:55.540627 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:55.540283    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vwcc9"
Jul 07 06:32:55.542073 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:55.542043    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5886655f8b-qm2fn" event={"ID":"feddeb52-1647-49c2-98ab-7cfdfade2460","Type":"ContainerStarted","Data":"999db51b821d5155fe47b19bced361a85e217db1fe1ac209cdd21e91a00c3e93"}
Jul 07 06:32:55.542209 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:55.542093    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5886655f8b-qm2fn" event={"ID":"feddeb52-1647-49c2-98ab-7cfdfade2460","Type":"ContainerStarted","Data":"d018dbf9919e25a4f590db314a6463cadbf755eadb42fff694870e829149a9f6"}
Jul 07 06:32:55.546212 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:55.546188    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vwcc9"
Jul 07 06:32:55.559298 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:55.559248    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vwcc9" podStartSLOduration=3.20333674 podStartE2EDuration="4.559226352s" totalImagesPullingTime="1.355889612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:32:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:32:55.55824259 +0000 UTC m=+58.922439137" watchObservedRunningTime="2026-07-07 06:32:55.559226352 +0000 UTC m=+58.923422898"
Jul 07 06:32:55.577478 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:55.577428    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5886655f8b-qm2fn" podStartSLOduration=1.577413504 podStartE2EDuration="1.577413504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:32:55.576259591 +0000 UTC m=+58.940456136" watchObservedRunningTime="2026-07-07 06:32:55.577413504 +0000 UTC m=+58.941610026"
Jul 07 06:32:55.779245 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:55.779203    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-kjcbn"]
Jul 07 06:32:55.782232 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:55.782209    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjcbn"
Jul 07 06:32:55.784661 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:55.784615    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 07 06:32:55.784768 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:55.784663    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-t2d9d\""
Jul 07 06:32:55.784768 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:55.784719    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 07 06:32:55.784768 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:55.784731    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 07 06:32:55.784768 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:55.784663    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 07 06:32:55.785026 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:55.785001    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 07 06:32:55.793001 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:55.792916    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-kjcbn"]
Jul 07 06:32:55.901854 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:55.901812    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c81bc8ba-1e33-4641-88fb-486835eaa393-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-kjcbn\" (UID: \"c81bc8ba-1e33-4641-88fb-486835eaa393\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjcbn"
Jul 07 06:32:55.902063 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:55.901883    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c81bc8ba-1e33-4641-88fb-486835eaa393-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-kjcbn\" (UID: \"c81bc8ba-1e33-4641-88fb-486835eaa393\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjcbn"
Jul 07 06:32:55.902063 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:55.902035    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tvj8\" (UniqueName: \"kubernetes.io/projected/c81bc8ba-1e33-4641-88fb-486835eaa393-kube-api-access-6tvj8\") pod \"prometheus-operator-567f7b8b7c-kjcbn\" (UID: \"c81bc8ba-1e33-4641-88fb-486835eaa393\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjcbn"
Jul 07 06:32:55.902194 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:55.902088    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c81bc8ba-1e33-4641-88fb-486835eaa393-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-kjcbn\" (UID: \"c81bc8ba-1e33-4641-88fb-486835eaa393\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjcbn"
Jul 07 06:32:56.003041 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:56.002997    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tvj8\" (UniqueName: \"kubernetes.io/projected/c81bc8ba-1e33-4641-88fb-486835eaa393-kube-api-access-6tvj8\") pod \"prometheus-operator-567f7b8b7c-kjcbn\" (UID: \"c81bc8ba-1e33-4641-88fb-486835eaa393\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjcbn"
Jul 07 06:32:56.003273 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:56.003057    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c81bc8ba-1e33-4641-88fb-486835eaa393-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-kjcbn\" (UID: \"c81bc8ba-1e33-4641-88fb-486835eaa393\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjcbn"
Jul 07 06:32:56.003273 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:56.003127    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c81bc8ba-1e33-4641-88fb-486835eaa393-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-kjcbn\" (UID: \"c81bc8ba-1e33-4641-88fb-486835eaa393\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjcbn"
Jul 07 06:32:56.003273 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:56.003187    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c81bc8ba-1e33-4641-88fb-486835eaa393-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-kjcbn\" (UID: \"c81bc8ba-1e33-4641-88fb-486835eaa393\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjcbn"
Jul 07 06:32:56.004120 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:56.004031    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c81bc8ba-1e33-4641-88fb-486835eaa393-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-kjcbn\" (UID: \"c81bc8ba-1e33-4641-88fb-486835eaa393\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjcbn"
Jul 07 06:32:56.006317 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:56.006283    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c81bc8ba-1e33-4641-88fb-486835eaa393-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-kjcbn\" (UID: \"c81bc8ba-1e33-4641-88fb-486835eaa393\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjcbn"
Jul 07 06:32:56.006494 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:56.006453    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c81bc8ba-1e33-4641-88fb-486835eaa393-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-kjcbn\" (UID: \"c81bc8ba-1e33-4641-88fb-486835eaa393\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjcbn"
Jul 07 06:32:56.012385 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:56.012349    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tvj8\" (UniqueName: \"kubernetes.io/projected/c81bc8ba-1e33-4641-88fb-486835eaa393-kube-api-access-6tvj8\") pod \"prometheus-operator-567f7b8b7c-kjcbn\" (UID: \"c81bc8ba-1e33-4641-88fb-486835eaa393\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjcbn"
Jul 07 06:32:56.031091 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:56.031055    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f45b5f89f-t2pnz"
Jul 07 06:32:56.031275 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:56.031137    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f45b5f89f-t2pnz"
Jul 07 06:32:56.032803 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:56.032770    2563 patch_prober.go:28] interesting pod/console-7f45b5f89f-t2pnz container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.9:8443/health\": dial tcp 10.133.0.9:8443: connect: connection refused" start-of-body=
Jul 07 06:32:56.033007 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:56.032842    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-7f45b5f89f-t2pnz" podUID="b6427e28-9acc-4588-ac81-a9c40bdee78e" containerName="console" probeResult="failure" output="Get \"https://10.133.0.9:8443/health\": dial tcp 10.133.0.9:8443: connect: connection refused"
Jul 07 06:32:56.092934 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:56.092898    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjcbn"
Jul 07 06:32:56.962318 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:56.962277    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-kjcbn"]
Jul 07 06:32:56.965080 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:32:56.965046    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81bc8ba_1e33_4641_88fb_486835eaa393.slice/crio-7f2ce6f9483fa2a006032a33b924d6ac6f01ace49a91356ccdb4b1093f3d677c WatchSource:0}: Error finding container 7f2ce6f9483fa2a006032a33b924d6ac6f01ace49a91356ccdb4b1093f3d677c: Status 404 returned error can't find the container with id 7f2ce6f9483fa2a006032a33b924d6ac6f01ace49a91356ccdb4b1093f3d677c
Jul 07 06:32:57.552095 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:57.552054    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vhq4w" event={"ID":"e5f0669f-f5b0-4d36-b791-681380b94eb6","Type":"ContainerStarted","Data":"acd60e5efb89388406df0cb3ff96fde266f7f460e0333b02d52649b6ae69bd60"}
Jul 07 06:32:57.553233 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:57.553211    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjcbn" event={"ID":"c81bc8ba-1e33-4641-88fb-486835eaa393","Type":"ContainerStarted","Data":"7f2ce6f9483fa2a006032a33b924d6ac6f01ace49a91356ccdb4b1093f3d677c"}
Jul 07 06:32:57.570537 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:57.570481    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vhq4w" podStartSLOduration=1.934624758 podStartE2EDuration="12.570460574s" totalImagesPullingTime="10.635835816s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 06:32:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:32:57.570062692 +0000 UTC m=+60.934259248" watchObservedRunningTime="2026-07-07 06:32:57.570460574 +0000 UTC m=+60.934657121"
Jul 07 06:32:59.463055 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:59.463014    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wnljf"
Jul 07 06:32:59.560935 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:59.560891    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjcbn" event={"ID":"c81bc8ba-1e33-4641-88fb-486835eaa393","Type":"ContainerStarted","Data":"50c329665540f4f3054528d68191ed7edf46a0803b3cbe870a7a1038dfd8b1f0"}
Jul 07 06:32:59.560935 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:59.560948    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjcbn" event={"ID":"c81bc8ba-1e33-4641-88fb-486835eaa393","Type":"ContainerStarted","Data":"8b45b58986e517f35a18d50c9ea856a3d8672bf01e80215937daefaa1ec10d7f"}
Jul 07 06:32:59.580676 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:32:59.580590    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjcbn" podStartSLOduration=2.560609647 podStartE2EDuration="4.580567968s" totalImagesPullingTime="2.019958321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:32:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:32:59.579428338 +0000 UTC m=+62.943624907" watchObservedRunningTime="2026-07-07 06:32:59.580567968 +0000 UTC m=+62.944764515"
Jul 07 06:33:01.114698 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.114662    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-9d268"]
Jul 07 06:33:01.119102 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.119077    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9d268"
Jul 07 06:33:01.123464 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.123439    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 07 06:33:01.123766 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.123743    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 07 06:33:01.123912 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.123819    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-45xjm\""
Jul 07 06:33:01.128640 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.128612    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-9d268"]
Jul 07 06:33:01.133364 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.133339    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2h7jr"]
Jul 07 06:33:01.136715 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.136697    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.141618 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.141591    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 07 06:33:01.141761 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.141662    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8tnd9\""
Jul 07 06:33:01.142320 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.142266    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 07 06:33:01.142320 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.142315    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 07 06:33:01.249054 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.249020    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a17275c3-428d-44ef-8207-0e1ea72271d4-node-exporter-textfile\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.249272 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.249090    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a17275c3-428d-44ef-8207-0e1ea72271d4-metrics-client-ca\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.249272 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.249121    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a17275c3-428d-44ef-8207-0e1ea72271d4-node-exporter-accelerators-collector-config\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.249272 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.249153    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6669dd2-240f-4113-9b5e-0aaa41741cca-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-9d268\" (UID: \"c6669dd2-240f-4113-9b5e-0aaa41741cca\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9d268"
Jul 07 06:33:01.249272 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.249174    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtwl2\" (UniqueName: \"kubernetes.io/projected/c6669dd2-240f-4113-9b5e-0aaa41741cca-kube-api-access-qtwl2\") pod \"openshift-state-metrics-54cf96dfcf-9d268\" (UID: \"c6669dd2-240f-4113-9b5e-0aaa41741cca\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9d268"
Jul 07 06:33:01.249272 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.249200    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a17275c3-428d-44ef-8207-0e1ea72271d4-sys\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.249272 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.249225    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a17275c3-428d-44ef-8207-0e1ea72271d4-node-exporter-wtmp\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.249272 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.249247    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a17275c3-428d-44ef-8207-0e1ea72271d4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.249272 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.249268    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdrvx\" (UniqueName: \"kubernetes.io/projected/a17275c3-428d-44ef-8207-0e1ea72271d4-kube-api-access-vdrvx\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.249613 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.249337    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6669dd2-240f-4113-9b5e-0aaa41741cca-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-9d268\" (UID: \"c6669dd2-240f-4113-9b5e-0aaa41741cca\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9d268"
Jul 07 06:33:01.249613 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.249431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a17275c3-428d-44ef-8207-0e1ea72271d4-root\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.249613 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.249461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a17275c3-428d-44ef-8207-0e1ea72271d4-node-exporter-tls\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.249613 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.249498    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6669dd2-240f-4113-9b5e-0aaa41741cca-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-9d268\" (UID: \"c6669dd2-240f-4113-9b5e-0aaa41741cca\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9d268"
Jul 07 06:33:01.350676 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.350614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6669dd2-240f-4113-9b5e-0aaa41741cca-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-9d268\" (UID: \"c6669dd2-240f-4113-9b5e-0aaa41741cca\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9d268"
Jul 07 06:33:01.350676 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.350675    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtwl2\" (UniqueName: \"kubernetes.io/projected/c6669dd2-240f-4113-9b5e-0aaa41741cca-kube-api-access-qtwl2\") pod \"openshift-state-metrics-54cf96dfcf-9d268\" (UID: \"c6669dd2-240f-4113-9b5e-0aaa41741cca\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9d268"
Jul 07 06:33:01.350897 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.350700    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a17275c3-428d-44ef-8207-0e1ea72271d4-sys\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.350897 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.350717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a17275c3-428d-44ef-8207-0e1ea72271d4-node-exporter-wtmp\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.350897 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.350736    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a17275c3-428d-44ef-8207-0e1ea72271d4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.350897 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.350752    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdrvx\" (UniqueName: \"kubernetes.io/projected/a17275c3-428d-44ef-8207-0e1ea72271d4-kube-api-access-vdrvx\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.350897 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.350780    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6669dd2-240f-4113-9b5e-0aaa41741cca-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-9d268\" (UID: \"c6669dd2-240f-4113-9b5e-0aaa41741cca\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9d268"
Jul 07 06:33:01.350897 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.350816    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a17275c3-428d-44ef-8207-0e1ea72271d4-sys\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.350897 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.350829    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a17275c3-428d-44ef-8207-0e1ea72271d4-root\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.350897 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.350855    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a17275c3-428d-44ef-8207-0e1ea72271d4-node-exporter-tls\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.350897 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.350893    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6669dd2-240f-4113-9b5e-0aaa41741cca-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-9d268\" (UID: \"c6669dd2-240f-4113-9b5e-0aaa41741cca\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9d268"
Jul 07 06:33:01.351366 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.350917    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a17275c3-428d-44ef-8207-0e1ea72271d4-node-exporter-textfile\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.351366 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.350969    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a17275c3-428d-44ef-8207-0e1ea72271d4-metrics-client-ca\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.351366 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.351001    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a17275c3-428d-44ef-8207-0e1ea72271d4-node-exporter-accelerators-collector-config\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.351366 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:33:01.351008    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 07 06:33:01.351366 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:33:01.351078    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a17275c3-428d-44ef-8207-0e1ea72271d4-node-exporter-tls podName:a17275c3-428d-44ef-8207-0e1ea72271d4 nodeName:}" failed. No retries permitted until 2026-07-07 06:33:01.851058211 +0000 UTC m=+65.215254748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a17275c3-428d-44ef-8207-0e1ea72271d4-node-exporter-tls") pod "node-exporter-2h7jr" (UID: "a17275c3-428d-44ef-8207-0e1ea72271d4") : secret "node-exporter-tls" not found
Jul 07 06:33:01.351366 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.351011    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a17275c3-428d-44ef-8207-0e1ea72271d4-root\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.351366 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.351317    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a17275c3-428d-44ef-8207-0e1ea72271d4-node-exporter-textfile\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.351366 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.351353    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a17275c3-428d-44ef-8207-0e1ea72271d4-node-exporter-wtmp\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.351963 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.351682    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6669dd2-240f-4113-9b5e-0aaa41741cca-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-9d268\" (UID: \"c6669dd2-240f-4113-9b5e-0aaa41741cca\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9d268"
Jul 07 06:33:01.351963 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.351784    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a17275c3-428d-44ef-8207-0e1ea72271d4-metrics-client-ca\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.351963 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.351807    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a17275c3-428d-44ef-8207-0e1ea72271d4-node-exporter-accelerators-collector-config\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.353302 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.353281    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a17275c3-428d-44ef-8207-0e1ea72271d4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.353403 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.353356    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6669dd2-240f-4113-9b5e-0aaa41741cca-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-9d268\" (UID: \"c6669dd2-240f-4113-9b5e-0aaa41741cca\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9d268"
Jul 07 06:33:01.353438 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.353417    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6669dd2-240f-4113-9b5e-0aaa41741cca-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-9d268\" (UID: \"c6669dd2-240f-4113-9b5e-0aaa41741cca\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9d268"
Jul 07 06:33:01.365200 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.363564    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdrvx\" (UniqueName: \"kubernetes.io/projected/a17275c3-428d-44ef-8207-0e1ea72271d4-kube-api-access-vdrvx\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.365200 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.363797    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtwl2\" (UniqueName: \"kubernetes.io/projected/c6669dd2-240f-4113-9b5e-0aaa41741cca-kube-api-access-qtwl2\") pod \"openshift-state-metrics-54cf96dfcf-9d268\" (UID: \"c6669dd2-240f-4113-9b5e-0aaa41741cca\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9d268"
Jul 07 06:33:01.428695 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.428612    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9d268"
Jul 07 06:33:01.574186 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.574151    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-9d268"]
Jul 07 06:33:01.577150 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:33:01.577116    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6669dd2_240f_4113_9b5e_0aaa41741cca.slice/crio-b6373b1f1941664397ccb267d476bffbf152a653f3f4d5ae8f23fccc9afdb58f WatchSource:0}: Error finding container b6373b1f1941664397ccb267d476bffbf152a653f3f4d5ae8f23fccc9afdb58f: Status 404 returned error can't find the container with id b6373b1f1941664397ccb267d476bffbf152a653f3f4d5ae8f23fccc9afdb58f
Jul 07 06:33:01.856264 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.856224    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a17275c3-428d-44ef-8207-0e1ea72271d4-node-exporter-tls\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:01.858777 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:01.858745    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a17275c3-428d-44ef-8207-0e1ea72271d4-node-exporter-tls\") pod \"node-exporter-2h7jr\" (UID: \"a17275c3-428d-44ef-8207-0e1ea72271d4\") " pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:02.046104 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:02.045941    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2h7jr"
Jul 07 06:33:02.060159 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:33:02.060115    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda17275c3_428d_44ef_8207_0e1ea72271d4.slice/crio-2561ae7c96b8522148cee47dc993d6d6aca36b24c623caf9289c0549bca0e48f WatchSource:0}: Error finding container 2561ae7c96b8522148cee47dc993d6d6aca36b24c623caf9289c0549bca0e48f: Status 404 returned error can't find the container with id 2561ae7c96b8522148cee47dc993d6d6aca36b24c623caf9289c0549bca0e48f
Jul 07 06:33:02.572696 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:02.572655    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2h7jr" event={"ID":"a17275c3-428d-44ef-8207-0e1ea72271d4","Type":"ContainerStarted","Data":"2561ae7c96b8522148cee47dc993d6d6aca36b24c623caf9289c0549bca0e48f"}
Jul 07 06:33:02.574563 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:02.574534    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9d268" event={"ID":"c6669dd2-240f-4113-9b5e-0aaa41741cca","Type":"ContainerStarted","Data":"0873af3ce89c30691d20a0b4949d05017d3b643de5bb9f48e84d146e67be3b60"}
Jul 07 06:33:02.574721 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:02.574588    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9d268" event={"ID":"c6669dd2-240f-4113-9b5e-0aaa41741cca","Type":"ContainerStarted","Data":"d3f8da065f8602774ef8e54f357ebd97ac86b66bc07451dfeb3c1542fc9c4cb6"}
Jul 07 06:33:02.574721 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:02.574608    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9d268" event={"ID":"c6669dd2-240f-4113-9b5e-0aaa41741cca","Type":"ContainerStarted","Data":"b6373b1f1941664397ccb267d476bffbf152a653f3f4d5ae8f23fccc9afdb58f"}
Jul 07 06:33:02.866124 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:02.866022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee9fd62f-81cf-4544-8454-9fa9f53c0768-original-pull-secret\") pod \"global-pull-secret-syncer-57vh8\" (UID: \"ee9fd62f-81cf-4544-8454-9fa9f53c0768\") " pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:33:02.866314 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:02.866151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c1e103c-f156-4b8a-8426-f18974595c51-metrics-certs\") pod \"network-metrics-daemon-mx8qb\" (UID: \"7c1e103c-f156-4b8a-8426-f18974595c51\") " pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:33:02.868494 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:02.868465    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 06:33:02.868657 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:02.868514    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 06:33:02.880042 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:02.880014    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c1e103c-f156-4b8a-8426-f18974595c51-metrics-certs\") pod \"network-metrics-daemon-mx8qb\" (UID: \"7c1e103c-f156-4b8a-8426-f18974595c51\") " pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:33:02.880198 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:02.880083    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee9fd62f-81cf-4544-8454-9fa9f53c0768-original-pull-secret\") pod \"global-pull-secret-syncer-57vh8\" (UID: \"ee9fd62f-81cf-4544-8454-9fa9f53c0768\") " pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:33:02.977668 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:02.977604    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5wsxr\""
Jul 07 06:33:02.986607 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:02.986558    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx8qb"
Jul 07 06:33:03.068009 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:03.067978    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47ljz\" (UniqueName: \"kubernetes.io/projected/ffb473f1-6af6-40d5-b15b-23939d3eff20-kube-api-access-47ljz\") pod \"network-check-target-jlprw\" (UID: \"ffb473f1-6af6-40d5-b15b-23939d3eff20\") " pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:33:03.070508 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:03.070479    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 06:33:03.080629 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:03.080595    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 06:33:03.091013 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:03.090982    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47ljz\" (UniqueName: \"kubernetes.io/projected/ffb473f1-6af6-40d5-b15b-23939d3eff20-kube-api-access-47ljz\") pod \"network-check-target-jlprw\" (UID: \"ffb473f1-6af6-40d5-b15b-23939d3eff20\") " pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:33:03.096901 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:03.096626    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dsscv\""
Jul 07 06:33:03.102127 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:03.102100    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57vh8"
Jul 07 06:33:03.103236 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:03.103138    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:33:03.153239 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:03.153153    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mx8qb"]
Jul 07 06:33:03.296486 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:33:03.296452    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1e103c_f156_4b8a_8426_f18974595c51.slice/crio-881b610bf3659ef12b552c77658762659d6d464f26256f0e2877c26c2f44ce8d WatchSource:0}: Error finding container 881b610bf3659ef12b552c77658762659d6d464f26256f0e2877c26c2f44ce8d: Status 404 returned error can't find the container with id 881b610bf3659ef12b552c77658762659d6d464f26256f0e2877c26c2f44ce8d
Jul 07 06:33:03.436906 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:03.436748    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jlprw"]
Jul 07 06:33:03.439700 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:33:03.439666    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb473f1_6af6_40d5_b15b_23939d3eff20.slice/crio-a3221086b6fc8982fd712037ab18bf8b9cf23ad4848355adca2c182615844080 WatchSource:0}: Error finding container a3221086b6fc8982fd712037ab18bf8b9cf23ad4848355adca2c182615844080: Status 404 returned error can't find the container with id a3221086b6fc8982fd712037ab18bf8b9cf23ad4848355adca2c182615844080
Jul 07 06:33:03.449862 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:03.449835    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-57vh8"]
Jul 07 06:33:03.452964 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:33:03.452934    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee9fd62f_81cf_4544_8454_9fa9f53c0768.slice/crio-a553543b7040c233c5f2c8a15585c1533f6d51cc762f05cfc64c4236c5b99f59 WatchSource:0}: Error finding container a553543b7040c233c5f2c8a15585c1533f6d51cc762f05cfc64c4236c5b99f59: Status 404 returned error can't find the container with id a553543b7040c233c5f2c8a15585c1533f6d51cc762f05cfc64c4236c5b99f59
Jul 07 06:33:03.579150 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:03.579118    2563 generic.go:356] "Generic (PLEG): container finished" podID="a17275c3-428d-44ef-8207-0e1ea72271d4" containerID="2e53e4bec21eb5d432e0ecf7c2d4b8462da376d57526e12c20e754ad2c6a2c29" exitCode=0
Jul 07 06:33:03.579608 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:03.579208    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2h7jr" event={"ID":"a17275c3-428d-44ef-8207-0e1ea72271d4","Type":"ContainerDied","Data":"2e53e4bec21eb5d432e0ecf7c2d4b8462da376d57526e12c20e754ad2c6a2c29"}
Jul 07 06:33:03.581508 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:03.581485    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9d268" event={"ID":"c6669dd2-240f-4113-9b5e-0aaa41741cca","Type":"ContainerStarted","Data":"8bdd1bd5c3b412ce891c054a139c64c6e8f5479630d7b26bb8c639f2253b894d"}
Jul 07 06:33:03.582783 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:03.582759    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jlprw" event={"ID":"ffb473f1-6af6-40d5-b15b-23939d3eff20","Type":"ContainerStarted","Data":"a3221086b6fc8982fd712037ab18bf8b9cf23ad4848355adca2c182615844080"}
Jul 07 06:33:03.584010 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:03.583990    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-57vh8" event={"ID":"ee9fd62f-81cf-4544-8454-9fa9f53c0768","Type":"ContainerStarted","Data":"a553543b7040c233c5f2c8a15585c1533f6d51cc762f05cfc64c4236c5b99f59"}
Jul 07 06:33:03.585096 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:03.585076    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mx8qb" event={"ID":"7c1e103c-f156-4b8a-8426-f18974595c51","Type":"ContainerStarted","Data":"881b610bf3659ef12b552c77658762659d6d464f26256f0e2877c26c2f44ce8d"}
Jul 07 06:33:03.620292 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:03.620228    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-9d268" podStartSLOduration=1.408277767 podStartE2EDuration="2.62020739s" totalImagesPullingTime="1.211929623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:33:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:33:03.619301448 +0000 UTC m=+66.983498021" watchObservedRunningTime="2026-07-07 06:33:03.62020739 +0000 UTC m=+66.984403936"
Jul 07 06:33:04.544744 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:04.544711    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gvkjl"
Jul 07 06:33:04.597781 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:04.597114    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2h7jr" event={"ID":"a17275c3-428d-44ef-8207-0e1ea72271d4","Type":"ContainerStarted","Data":"43874d8b31b0ee68324ee6ce48231d0cb746b2bda1dadd32e42c2f700ddd2693"}
Jul 07 06:33:04.597781 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:04.597238    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2h7jr" event={"ID":"a17275c3-428d-44ef-8207-0e1ea72271d4","Type":"ContainerStarted","Data":"4cbc5d7cc94f09a5638b9454d64bff9fb67f5696c1eb7878a5046c4a637cb44c"}
Jul 07 06:33:04.619233 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:04.618993    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2h7jr" podStartSLOduration=2.279505038 podStartE2EDuration="3.618970312s" totalImagesPullingTime="1.263376631s" totalInitContainerRuntime="76.088643ms" isStatefulPod=true podCreationTimestamp="2026-07-07 06:33:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:33:04.617452773 +0000 UTC m=+67.981649341" watchObservedRunningTime="2026-07-07 06:33:04.618970312 +0000 UTC m=+67.983166860"
Jul 07 06:33:04.797080 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:04.796253    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5886655f8b-qm2fn"
Jul 07 06:33:04.797080 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:04.796327    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5886655f8b-qm2fn"
Jul 07 06:33:04.797749 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:04.797711    2563 patch_prober.go:28] interesting pod/console-5886655f8b-qm2fn container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.11:8443/health\": dial tcp 10.133.0.11:8443: connect: connection refused" start-of-body=
Jul 07 06:33:04.797837 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:04.797771    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5886655f8b-qm2fn" podUID="feddeb52-1647-49c2-98ab-7cfdfade2460" containerName="console" probeResult="failure" output="Get \"https://10.133.0.11:8443/health\": dial tcp 10.133.0.11:8443: connect: connection refused"
Jul 07 06:33:05.524479 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.524440    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-f7646d95-nsbd6"]
Jul 07 06:33:05.532872 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.532833    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f7646d95-nsbd6"
Jul 07 06:33:05.534896 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.534864    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 07 06:33:05.535281 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.535259    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-7mgh5\""
Jul 07 06:33:05.535281 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.535263    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 07 06:33:05.535539 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.535428    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4870j32g6pltf\""
Jul 07 06:33:05.535659 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.535540    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 07 06:33:05.536196 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.535707    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 07 06:33:05.538444 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.538415    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f7646d95-nsbd6"]
Jul 07 06:33:05.590753 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.590715    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/45ef3bb7-96c9-4d6c-8214-093409d1856e-secret-metrics-server-tls\") pod \"metrics-server-f7646d95-nsbd6\" (UID: \"45ef3bb7-96c9-4d6c-8214-093409d1856e\") " pod="openshift-monitoring/metrics-server-f7646d95-nsbd6"
Jul 07 06:33:05.590753 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.590753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/45ef3bb7-96c9-4d6c-8214-093409d1856e-client-ca-bundle\") pod \"metrics-server-f7646d95-nsbd6\" (UID: \"45ef3bb7-96c9-4d6c-8214-093409d1856e\") " pod="openshift-monitoring/metrics-server-f7646d95-nsbd6"
Jul 07 06:33:05.590943 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.590779    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/45ef3bb7-96c9-4d6c-8214-093409d1856e-audit-log\") pod \"metrics-server-f7646d95-nsbd6\" (UID: \"45ef3bb7-96c9-4d6c-8214-093409d1856e\") " pod="openshift-monitoring/metrics-server-f7646d95-nsbd6"
Jul 07 06:33:05.590943 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.590845    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2fnh\" (UniqueName: \"kubernetes.io/projected/45ef3bb7-96c9-4d6c-8214-093409d1856e-kube-api-access-j2fnh\") pod \"metrics-server-f7646d95-nsbd6\" (UID: \"45ef3bb7-96c9-4d6c-8214-093409d1856e\") " pod="openshift-monitoring/metrics-server-f7646d95-nsbd6"
Jul 07 06:33:05.590943 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.590918    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/45ef3bb7-96c9-4d6c-8214-093409d1856e-metrics-server-audit-profiles\") pod \"metrics-server-f7646d95-nsbd6\" (UID: \"45ef3bb7-96c9-4d6c-8214-093409d1856e\") " pod="openshift-monitoring/metrics-server-f7646d95-nsbd6"
Jul 07 06:33:05.591095 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.590958    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/45ef3bb7-96c9-4d6c-8214-093409d1856e-secret-metrics-server-client-certs\") pod \"metrics-server-f7646d95-nsbd6\" (UID: \"45ef3bb7-96c9-4d6c-8214-093409d1856e\") " pod="openshift-monitoring/metrics-server-f7646d95-nsbd6"
Jul 07 06:33:05.591095 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.590992    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ef3bb7-96c9-4d6c-8214-093409d1856e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f7646d95-nsbd6\" (UID: \"45ef3bb7-96c9-4d6c-8214-093409d1856e\") " pod="openshift-monitoring/metrics-server-f7646d95-nsbd6"
Jul 07 06:33:05.604755 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.604718    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mx8qb" event={"ID":"7c1e103c-f156-4b8a-8426-f18974595c51","Type":"ContainerStarted","Data":"d93d486789014eb84d4d67b3836b6d3994c7fb5eb6c56b437bb001cc6584e14f"}
Jul 07 06:33:05.605309 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.604782    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mx8qb" event={"ID":"7c1e103c-f156-4b8a-8426-f18974595c51","Type":"ContainerStarted","Data":"aa1a06f5abf5bf5fb9cd7fe7d47f95d5895cb3bf921a29fc36e70583300da847"}
Jul 07 06:33:05.623923 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.623865    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mx8qb" podStartSLOduration=67.00868444 podStartE2EDuration="1m8.623846539s" totalImagesPullingTime="1.615162099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:33:05.622335488 +0000 UTC m=+68.986532061" watchObservedRunningTime="2026-07-07 06:33:05.623846539 +0000 UTC m=+68.988043161"
Jul 07 06:33:05.691761 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.691723    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/45ef3bb7-96c9-4d6c-8214-093409d1856e-secret-metrics-server-tls\") pod \"metrics-server-f7646d95-nsbd6\" (UID: \"45ef3bb7-96c9-4d6c-8214-093409d1856e\") " pod="openshift-monitoring/metrics-server-f7646d95-nsbd6"
Jul 07 06:33:05.691761 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.691767    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/45ef3bb7-96c9-4d6c-8214-093409d1856e-client-ca-bundle\") pod \"metrics-server-f7646d95-nsbd6\" (UID: \"45ef3bb7-96c9-4d6c-8214-093409d1856e\") " pod="openshift-monitoring/metrics-server-f7646d95-nsbd6"
Jul 07 06:33:05.692032 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.691794    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/45ef3bb7-96c9-4d6c-8214-093409d1856e-audit-log\") pod \"metrics-server-f7646d95-nsbd6\" (UID: \"45ef3bb7-96c9-4d6c-8214-093409d1856e\") " pod="openshift-monitoring/metrics-server-f7646d95-nsbd6"
Jul 07 06:33:05.692032 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.691825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2fnh\" (UniqueName: \"kubernetes.io/projected/45ef3bb7-96c9-4d6c-8214-093409d1856e-kube-api-access-j2fnh\") pod \"metrics-server-f7646d95-nsbd6\" (UID: \"45ef3bb7-96c9-4d6c-8214-093409d1856e\") " pod="openshift-monitoring/metrics-server-f7646d95-nsbd6"
Jul 07 06:33:05.692032 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.691906    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/45ef3bb7-96c9-4d6c-8214-093409d1856e-metrics-server-audit-profiles\") pod \"metrics-server-f7646d95-nsbd6\" (UID: \"45ef3bb7-96c9-4d6c-8214-093409d1856e\") " pod="openshift-monitoring/metrics-server-f7646d95-nsbd6"
Jul 07 06:33:05.692032 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.691936    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/45ef3bb7-96c9-4d6c-8214-093409d1856e-secret-metrics-server-client-certs\") pod \"metrics-server-f7646d95-nsbd6\" (UID: \"45ef3bb7-96c9-4d6c-8214-093409d1856e\") " pod="openshift-monitoring/metrics-server-f7646d95-nsbd6"
Jul 07 06:33:05.692032 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.691966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ef3bb7-96c9-4d6c-8214-093409d1856e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f7646d95-nsbd6\" (UID: \"45ef3bb7-96c9-4d6c-8214-093409d1856e\") " pod="openshift-monitoring/metrics-server-f7646d95-nsbd6"
Jul 07 06:33:05.692871 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.692827    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ef3bb7-96c9-4d6c-8214-093409d1856e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f7646d95-nsbd6\" (UID: \"45ef3bb7-96c9-4d6c-8214-093409d1856e\") " pod="openshift-monitoring/metrics-server-f7646d95-nsbd6"
Jul 07 06:33:05.693166 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.692988    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/45ef3bb7-96c9-4d6c-8214-093409d1856e-audit-log\") pod \"metrics-server-f7646d95-nsbd6\" (UID: \"45ef3bb7-96c9-4d6c-8214-093409d1856e\") " pod="openshift-monitoring/metrics-server-f7646d95-nsbd6"
Jul 07 06:33:05.693166 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.693105    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/45ef3bb7-96c9-4d6c-8214-093409d1856e-metrics-server-audit-profiles\") pod \"metrics-server-f7646d95-nsbd6\" (UID: \"45ef3bb7-96c9-4d6c-8214-093409d1856e\") " pod="openshift-monitoring/metrics-server-f7646d95-nsbd6"
Jul 07 06:33:05.695044 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.695014    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/45ef3bb7-96c9-4d6c-8214-093409d1856e-client-ca-bundle\") pod \"metrics-server-f7646d95-nsbd6\" (UID: \"45ef3bb7-96c9-4d6c-8214-093409d1856e\") " pod="openshift-monitoring/metrics-server-f7646d95-nsbd6"
Jul 07 06:33:05.695162 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.695076    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/45ef3bb7-96c9-4d6c-8214-093409d1856e-secret-metrics-server-client-certs\") pod \"metrics-server-f7646d95-nsbd6\" (UID: \"45ef3bb7-96c9-4d6c-8214-093409d1856e\") " pod="openshift-monitoring/metrics-server-f7646d95-nsbd6"
Jul 07 06:33:05.695231 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.695197    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/45ef3bb7-96c9-4d6c-8214-093409d1856e-secret-metrics-server-tls\") pod \"metrics-server-f7646d95-nsbd6\" (UID: \"45ef3bb7-96c9-4d6c-8214-093409d1856e\") " pod="openshift-monitoring/metrics-server-f7646d95-nsbd6"
Jul 07 06:33:05.701736 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.701708    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2fnh\" (UniqueName: \"kubernetes.io/projected/45ef3bb7-96c9-4d6c-8214-093409d1856e-kube-api-access-j2fnh\") pod \"metrics-server-f7646d95-nsbd6\" (UID: \"45ef3bb7-96c9-4d6c-8214-093409d1856e\") " pod="openshift-monitoring/metrics-server-f7646d95-nsbd6"
Jul 07 06:33:05.847094 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.847047    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f7646d95-nsbd6"
Jul 07 06:33:05.910821 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.910579    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-p5kjp"]
Jul 07 06:33:05.944139 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.943593    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-p5kjp"]
Jul 07 06:33:05.944139 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.943669    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f45b5f89f-t2pnz"]
Jul 07 06:33:05.944139 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.943798    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-p5kjp"
Jul 07 06:33:05.953936 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.952778    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 07 06:33:05.953936 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.953748    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-ldg4l\""
Jul 07 06:33:05.958039 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.958010    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84b69f5449-7d49k"]
Jul 07 06:33:05.979561 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.979369    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84b69f5449-7d49k"]
Jul 07 06:33:05.979561 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.979500    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:05.995946 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.995445    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/26aef36b-c5d2-4c16-9bfc-917f6eba2bb9-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-p5kjp\" (UID: \"26aef36b-c5d2-4c16-9bfc-917f6eba2bb9\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-p5kjp"
Jul 07 06:33:05.995946 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:05.995502    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9gjg\" (UniqueName: \"kubernetes.io/projected/26aef36b-c5d2-4c16-9bfc-917f6eba2bb9-kube-api-access-s9gjg\") pod \"monitoring-plugin-d5dcff7cf-p5kjp\" (UID: \"26aef36b-c5d2-4c16-9bfc-917f6eba2bb9\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-p5kjp"
Jul 07 06:33:06.058202 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.057574    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f7646d95-nsbd6"]
Jul 07 06:33:06.096009 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.095972    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3eec4752-565d-4551-b768-371ad13fdc5b-service-ca\") pod \"console-84b69f5449-7d49k\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") " pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:06.096192 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.096021    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3eec4752-565d-4551-b768-371ad13fdc5b-console-config\") pod \"console-84b69f5449-7d49k\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") " pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:06.096192 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.096057    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3eec4752-565d-4551-b768-371ad13fdc5b-console-oauth-config\") pod \"console-84b69f5449-7d49k\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") " pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:06.096192 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.096082    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/26aef36b-c5d2-4c16-9bfc-917f6eba2bb9-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-p5kjp\" (UID: \"26aef36b-c5d2-4c16-9bfc-917f6eba2bb9\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-p5kjp"
Jul 07 06:33:06.096192 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.096127    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9gjg\" (UniqueName: \"kubernetes.io/projected/26aef36b-c5d2-4c16-9bfc-917f6eba2bb9-kube-api-access-s9gjg\") pod \"monitoring-plugin-d5dcff7cf-p5kjp\" (UID: \"26aef36b-c5d2-4c16-9bfc-917f6eba2bb9\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-p5kjp"
Jul 07 06:33:06.096192 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.096151    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3eec4752-565d-4551-b768-371ad13fdc5b-console-serving-cert\") pod \"console-84b69f5449-7d49k\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") " pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:06.096192 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.096177    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3eec4752-565d-4551-b768-371ad13fdc5b-oauth-serving-cert\") pod \"console-84b69f5449-7d49k\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") " pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:06.096472 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:33:06.096231    2563 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 07 06:33:06.096472 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.096255    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eec4752-565d-4551-b768-371ad13fdc5b-trusted-ca-bundle\") pod \"console-84b69f5449-7d49k\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") " pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:06.096472 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:33:06.096315    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26aef36b-c5d2-4c16-9bfc-917f6eba2bb9-monitoring-plugin-cert podName:26aef36b-c5d2-4c16-9bfc-917f6eba2bb9 nodeName:}" failed. No retries permitted until 2026-07-07 06:33:06.596291678 +0000 UTC m=+69.960488203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/26aef36b-c5d2-4c16-9bfc-917f6eba2bb9-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-p5kjp" (UID: "26aef36b-c5d2-4c16-9bfc-917f6eba2bb9") : secret "monitoring-plugin-cert" not found
Jul 07 06:33:06.096472 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.096381    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v9tn\" (UniqueName: \"kubernetes.io/projected/3eec4752-565d-4551-b768-371ad13fdc5b-kube-api-access-4v9tn\") pod \"console-84b69f5449-7d49k\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") " pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:06.107817 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.107736    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9gjg\" (UniqueName: \"kubernetes.io/projected/26aef36b-c5d2-4c16-9bfc-917f6eba2bb9-kube-api-access-s9gjg\") pod \"monitoring-plugin-d5dcff7cf-p5kjp\" (UID: \"26aef36b-c5d2-4c16-9bfc-917f6eba2bb9\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-p5kjp"
Jul 07 06:33:06.197275 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.197237    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3eec4752-565d-4551-b768-371ad13fdc5b-service-ca\") pod \"console-84b69f5449-7d49k\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") " pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:06.197472 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.197284    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3eec4752-565d-4551-b768-371ad13fdc5b-console-config\") pod \"console-84b69f5449-7d49k\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") " pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:06.197472 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.197322    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3eec4752-565d-4551-b768-371ad13fdc5b-console-oauth-config\") pod \"console-84b69f5449-7d49k\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") " pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:06.197472 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.197378    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3eec4752-565d-4551-b768-371ad13fdc5b-console-serving-cert\") pod \"console-84b69f5449-7d49k\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") " pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:06.197472 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.197403    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3eec4752-565d-4551-b768-371ad13fdc5b-oauth-serving-cert\") pod \"console-84b69f5449-7d49k\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") " pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:06.197472 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.197426    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eec4752-565d-4551-b768-371ad13fdc5b-trusted-ca-bundle\") pod \"console-84b69f5449-7d49k\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") " pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:06.197750 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.197502    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v9tn\" (UniqueName: \"kubernetes.io/projected/3eec4752-565d-4551-b768-371ad13fdc5b-kube-api-access-4v9tn\") pod \"console-84b69f5449-7d49k\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") " pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:06.199187 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.198509    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3eec4752-565d-4551-b768-371ad13fdc5b-service-ca\") pod \"console-84b69f5449-7d49k\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") " pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:06.199187 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.199130    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3eec4752-565d-4551-b768-371ad13fdc5b-oauth-serving-cert\") pod \"console-84b69f5449-7d49k\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") " pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:06.199683 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.199616    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3eec4752-565d-4551-b768-371ad13fdc5b-console-config\") pod \"console-84b69f5449-7d49k\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") " pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:06.199683 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.199622    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eec4752-565d-4551-b768-371ad13fdc5b-trusted-ca-bundle\") pod \"console-84b69f5449-7d49k\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") " pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:06.201628 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.201599    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3eec4752-565d-4551-b768-371ad13fdc5b-console-serving-cert\") pod \"console-84b69f5449-7d49k\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") " pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:06.201867 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.201846    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3eec4752-565d-4551-b768-371ad13fdc5b-console-oauth-config\") pod \"console-84b69f5449-7d49k\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") " pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:06.205930 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.205877    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v9tn\" (UniqueName: \"kubernetes.io/projected/3eec4752-565d-4551-b768-371ad13fdc5b-kube-api-access-4v9tn\") pod \"console-84b69f5449-7d49k\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") " pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:06.293913 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.293872    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:06.600567 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.600521    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/26aef36b-c5d2-4c16-9bfc-917f6eba2bb9-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-p5kjp\" (UID: \"26aef36b-c5d2-4c16-9bfc-917f6eba2bb9\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-p5kjp"
Jul 07 06:33:06.604415 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.604374    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/26aef36b-c5d2-4c16-9bfc-917f6eba2bb9-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-p5kjp\" (UID: \"26aef36b-c5d2-4c16-9bfc-917f6eba2bb9\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-p5kjp"
Jul 07 06:33:06.867729 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:06.867621    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-p5kjp"
Jul 07 06:33:07.155020 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:33:07.154942    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ef3bb7_96c9_4d6c_8214_093409d1856e.slice/crio-5903dd3baa048ae57c8ed38526efd779e66d70cbdb3879a98e6353ce2d3635db WatchSource:0}: Error finding container 5903dd3baa048ae57c8ed38526efd779e66d70cbdb3879a98e6353ce2d3635db: Status 404 returned error can't find the container with id 5903dd3baa048ae57c8ed38526efd779e66d70cbdb3879a98e6353ce2d3635db
Jul 07 06:33:07.419900 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.419779    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 06:33:07.425376 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.425340    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.430707 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.429206    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 07 06:33:07.430707 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.429511    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 07 06:33:07.430707 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.429724    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 07 06:33:07.430707 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.429914    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 07 06:33:07.430707 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.430299    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 07 06:33:07.430707 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.430484    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 07 06:33:07.431113 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.430882    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2rvufhnfgeecn\""
Jul 07 06:33:07.431167 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.431119    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 07 06:33:07.431167 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.431130    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qfsrh\""
Jul 07 06:33:07.431167 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.431160    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 07 06:33:07.431517 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.431473    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 07 06:33:07.431779 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.431747    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 07 06:33:07.431779 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.431768    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 07 06:33:07.432536 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.432511    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 07 06:33:07.433182 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.433161    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 07 06:33:07.435619 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.435593    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 07 06:33:07.440882 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.440842    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 06:33:07.509789 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.509744    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5f6429ad-0f44-4eaa-9638-0627dc4e392a-config\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.509962 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.509813    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5f6429ad-0f44-4eaa-9638-0627dc4e392a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.509962 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.509853    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5f6429ad-0f44-4eaa-9638-0627dc4e392a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.509962 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.509880    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5f6429ad-0f44-4eaa-9638-0627dc4e392a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.509962 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.509904    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5f6429ad-0f44-4eaa-9638-0627dc4e392a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.509962 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.509932    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f6429ad-0f44-4eaa-9638-0627dc4e392a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.510184 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.510039    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5f6429ad-0f44-4eaa-9638-0627dc4e392a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.510184 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.510072    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5f6429ad-0f44-4eaa-9638-0627dc4e392a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.510184 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.510087    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5f6429ad-0f44-4eaa-9638-0627dc4e392a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.510184 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.510116    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5f6429ad-0f44-4eaa-9638-0627dc4e392a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.510334 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.510192    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f6429ad-0f44-4eaa-9638-0627dc4e392a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.510334 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.510224    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6429ad-0f44-4eaa-9638-0627dc4e392a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.510334 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.510249    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f6429ad-0f44-4eaa-9638-0627dc4e392a-web-config\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.510334 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.510278    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f6429ad-0f44-4eaa-9638-0627dc4e392a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.510334 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.510303    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6429ad-0f44-4eaa-9638-0627dc4e392a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.510334 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.510326    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6429ad-0f44-4eaa-9638-0627dc4e392a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.510575 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.510351    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f6429ad-0f44-4eaa-9638-0627dc4e392a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.510575 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.510374    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbgzf\" (UniqueName: \"kubernetes.io/projected/5f6429ad-0f44-4eaa-9638-0627dc4e392a-kube-api-access-vbgzf\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.510575 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.510413    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f6429ad-0f44-4eaa-9638-0627dc4e392a-config-out\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.510575 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.510483    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5f6429ad-0f44-4eaa-9638-0627dc4e392a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.611097 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.611044    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5f6429ad-0f44-4eaa-9638-0627dc4e392a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.611097 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.611098    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5f6429ad-0f44-4eaa-9638-0627dc4e392a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.611339 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.611124    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5f6429ad-0f44-4eaa-9638-0627dc4e392a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.611339 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.611155    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5f6429ad-0f44-4eaa-9638-0627dc4e392a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.611339 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.611191    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f6429ad-0f44-4eaa-9638-0627dc4e392a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.611339 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.611218    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6429ad-0f44-4eaa-9638-0627dc4e392a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.611339 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.611243    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f6429ad-0f44-4eaa-9638-0627dc4e392a-web-config\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.611339 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.611271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f6429ad-0f44-4eaa-9638-0627dc4e392a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.611339 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.611293    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6429ad-0f44-4eaa-9638-0627dc4e392a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.611339 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.611316    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6429ad-0f44-4eaa-9638-0627dc4e392a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.611339 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.611341    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f6429ad-0f44-4eaa-9638-0627dc4e392a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.611771 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.611362    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbgzf\" (UniqueName: \"kubernetes.io/projected/5f6429ad-0f44-4eaa-9638-0627dc4e392a-kube-api-access-vbgzf\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.611771 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.611404    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f6429ad-0f44-4eaa-9638-0627dc4e392a-config-out\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.611771 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.611434    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5f6429ad-0f44-4eaa-9638-0627dc4e392a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.611771 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.611465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5f6429ad-0f44-4eaa-9638-0627dc4e392a-config\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.611771 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.611485    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5f6429ad-0f44-4eaa-9638-0627dc4e392a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.611771 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.611524    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5f6429ad-0f44-4eaa-9638-0627dc4e392a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.611771 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.611554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5f6429ad-0f44-4eaa-9638-0627dc4e392a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.611771 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.611575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5f6429ad-0f44-4eaa-9638-0627dc4e392a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.611771 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.611605    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f6429ad-0f44-4eaa-9638-0627dc4e392a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.612884 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.612854    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5f6429ad-0f44-4eaa-9638-0627dc4e392a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.613022 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.612958    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6429ad-0f44-4eaa-9638-0627dc4e392a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.614294 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.613578    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5f6429ad-0f44-4eaa-9638-0627dc4e392a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.614294 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.614178    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5f6429ad-0f44-4eaa-9638-0627dc4e392a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.615534 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.615509    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f6429ad-0f44-4eaa-9638-0627dc4e392a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.616927 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.616902    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f6429ad-0f44-4eaa-9638-0627dc4e392a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.617560 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.617538    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6429ad-0f44-4eaa-9638-0627dc4e392a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.618708 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.618541    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5f6429ad-0f44-4eaa-9638-0627dc4e392a-config\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.619478 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.619447    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5f6429ad-0f44-4eaa-9638-0627dc4e392a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.621484 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.621459    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f6429ad-0f44-4eaa-9638-0627dc4e392a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.622337 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.621586    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f6429ad-0f44-4eaa-9638-0627dc4e392a-config-out\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.622337 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.622308    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6429ad-0f44-4eaa-9638-0627dc4e392a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.622961 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.622935    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5f6429ad-0f44-4eaa-9638-0627dc4e392a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.624003 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.623875    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f7646d95-nsbd6" event={"ID":"45ef3bb7-96c9-4d6c-8214-093409d1856e","Type":"ContainerStarted","Data":"5903dd3baa048ae57c8ed38526efd779e66d70cbdb3879a98e6353ce2d3635db"}
Jul 07 06:33:07.625341 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.625299    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5f6429ad-0f44-4eaa-9638-0627dc4e392a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.625411 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.625369    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5f6429ad-0f44-4eaa-9638-0627dc4e392a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.625923 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.625900    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f6429ad-0f44-4eaa-9638-0627dc4e392a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.625923 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.625915    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f6429ad-0f44-4eaa-9638-0627dc4e392a-web-config\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.626436 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.626417    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbgzf\" (UniqueName: \"kubernetes.io/projected/5f6429ad-0f44-4eaa-9638-0627dc4e392a-kube-api-access-vbgzf\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.626436 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.626427    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5f6429ad-0f44-4eaa-9638-0627dc4e392a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.628700 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.628612    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5f6429ad-0f44-4eaa-9638-0627dc4e392a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5f6429ad-0f44-4eaa-9638-0627dc4e392a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:07.742773 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:07.742202    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:08.967822 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:08.967474    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-p5kjp"]
Jul 07 06:33:08.972230 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:33:08.972195    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26aef36b_c5d2_4c16_9bfc_917f6eba2bb9.slice/crio-98831f6d5782fc3b6c313f8c0f6b4aa3d48c8e2e1d7bec1688fa8f9b76539c76 WatchSource:0}: Error finding container 98831f6d5782fc3b6c313f8c0f6b4aa3d48c8e2e1d7bec1688fa8f9b76539c76: Status 404 returned error can't find the container with id 98831f6d5782fc3b6c313f8c0f6b4aa3d48c8e2e1d7bec1688fa8f9b76539c76
Jul 07 06:33:09.197584 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:09.197553    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84b69f5449-7d49k"]
Jul 07 06:33:09.200377 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:33:09.200346    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eec4752_565d_4551_b768_371ad13fdc5b.slice/crio-7ae1de435f07fc6a0199e6701a01385c88e25ce3c83cd9d530300c17538f2bda WatchSource:0}: Error finding container 7ae1de435f07fc6a0199e6701a01385c88e25ce3c83cd9d530300c17538f2bda: Status 404 returned error can't find the container with id 7ae1de435f07fc6a0199e6701a01385c88e25ce3c83cd9d530300c17538f2bda
Jul 07 06:33:09.210252 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:09.210109    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 06:33:09.213815 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:33:09.213748    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6429ad_0f44_4eaa_9638_0627dc4e392a.slice/crio-7f4b0aa8ecdc99415718ff30b432fca5d87ed60a6856344612677e140fcda174 WatchSource:0}: Error finding container 7f4b0aa8ecdc99415718ff30b432fca5d87ed60a6856344612677e140fcda174: Status 404 returned error can't find the container with id 7f4b0aa8ecdc99415718ff30b432fca5d87ed60a6856344612677e140fcda174
Jul 07 06:33:09.635239 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:09.635195    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jlprw" event={"ID":"ffb473f1-6af6-40d5-b15b-23939d3eff20","Type":"ContainerStarted","Data":"a9c995f2c0251fec3819f6afb3490545736671e2388651e83b28850fdb22d5f9"}
Jul 07 06:33:09.635425 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:09.635337    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:33:09.637480 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:09.637398    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b69f5449-7d49k" event={"ID":"3eec4752-565d-4551-b768-371ad13fdc5b","Type":"ContainerStarted","Data":"bf7cbd6fe2b15941a3ee417d4f6c6ece197e5a711bbc8aaabac9d5836e5cbd8c"}
Jul 07 06:33:09.637480 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:09.637445    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b69f5449-7d49k" event={"ID":"3eec4752-565d-4551-b768-371ad13fdc5b","Type":"ContainerStarted","Data":"7ae1de435f07fc6a0199e6701a01385c88e25ce3c83cd9d530300c17538f2bda"}
Jul 07 06:33:09.639409 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:09.639380    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f6429ad-0f44-4eaa-9638-0627dc4e392a","Type":"ContainerStarted","Data":"7f4b0aa8ecdc99415718ff30b432fca5d87ed60a6856344612677e140fcda174"}
Jul 07 06:33:09.640959 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:09.640909    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-p5kjp" event={"ID":"26aef36b-c5d2-4c16-9bfc-917f6eba2bb9","Type":"ContainerStarted","Data":"98831f6d5782fc3b6c313f8c0f6b4aa3d48c8e2e1d7bec1688fa8f9b76539c76"}
Jul 07 06:33:09.642676 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:09.642622    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-57vh8" event={"ID":"ee9fd62f-81cf-4544-8454-9fa9f53c0768","Type":"ContainerStarted","Data":"5b82cbf5877b954bd4d8a262cdd7ebb43e0102f009fdeefd027b383768ea63ad"}
Jul 07 06:33:09.650673 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:09.649993    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jlprw" podStartSLOduration=67.322391849 podStartE2EDuration="1m12.649968348s" totalImagesPullingTime="5.327576499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:33:09.648538727 +0000 UTC m=+73.012735292" watchObservedRunningTime="2026-07-07 06:33:09.649968348 +0000 UTC m=+73.014164894"
Jul 07 06:33:09.666875 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:09.666796    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84b69f5449-7d49k" podStartSLOduration=4.666777998 podStartE2EDuration="4.666777998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:33:09.666271266 +0000 UTC m=+73.030467811" watchObservedRunningTime="2026-07-07 06:33:09.666777998 +0000 UTC m=+73.030974543"
Jul 07 06:33:09.682293 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:09.681606    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-57vh8" podStartSLOduration=67.3505118 podStartE2EDuration="1m12.681584084s" totalImagesPullingTime="5.331072284s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 06:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:33:09.680552498 +0000 UTC m=+73.044749043" watchObservedRunningTime="2026-07-07 06:33:09.681584084 +0000 UTC m=+73.045780630"
Jul 07 06:33:10.660927 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:10.660884    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f7646d95-nsbd6" event={"ID":"45ef3bb7-96c9-4d6c-8214-093409d1856e","Type":"ContainerStarted","Data":"6a9153fad869ae85b81e4ed2b59609f01411fce1edca421258a0ca9ed842d3ae"}
Jul 07 06:33:10.773748 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:10.773688    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-f7646d95-nsbd6" podStartSLOduration=3.017882712 podStartE2EDuration="5.773666889s" totalImagesPullingTime="2.755784177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:33:10.679910743 +0000 UTC m=+74.044107289" watchObservedRunningTime="2026-07-07 06:33:10.773666889 +0000 UTC m=+74.137863432"
Jul 07 06:33:10.774723 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:10.774698    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5886655f8b-qm2fn"]
Jul 07 06:33:10.811329 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:10.808893    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f7b644f79-shxdr"]
Jul 07 06:33:10.812926 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:10.812803    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:33:10.822966 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:10.822917    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f7b644f79-shxdr"]
Jul 07 06:33:10.959011 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:10.958956    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-console-oauth-config\") pod \"console-5f7b644f79-shxdr\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") " pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:33:10.959173 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:10.959033    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-trusted-ca-bundle\") pod \"console-5f7b644f79-shxdr\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") " pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:33:10.959173 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:10.959085    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2292b\" (UniqueName: \"kubernetes.io/projected/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-kube-api-access-2292b\") pod \"console-5f7b644f79-shxdr\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") " pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:33:10.959173 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:10.959128    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-oauth-serving-cert\") pod \"console-5f7b644f79-shxdr\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") " pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:33:10.959341 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:10.959187    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-console-serving-cert\") pod \"console-5f7b644f79-shxdr\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") " pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:33:10.959341 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:10.959211    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-console-config\") pod \"console-5f7b644f79-shxdr\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") " pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:33:10.959341 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:10.959255    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-service-ca\") pod \"console-5f7b644f79-shxdr\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") " pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:33:11.060507 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:11.060459    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-trusted-ca-bundle\") pod \"console-5f7b644f79-shxdr\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") " pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:33:11.060507 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:11.060510    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2292b\" (UniqueName: \"kubernetes.io/projected/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-kube-api-access-2292b\") pod \"console-5f7b644f79-shxdr\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") " pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:33:11.060846 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:11.060538    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-oauth-serving-cert\") pod \"console-5f7b644f79-shxdr\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") " pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:33:11.060846 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:11.060575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-console-serving-cert\") pod \"console-5f7b644f79-shxdr\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") " pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:33:11.060846 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:11.060598    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-console-config\") pod \"console-5f7b644f79-shxdr\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") " pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:33:11.060846 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:11.060651    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-service-ca\") pod \"console-5f7b644f79-shxdr\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") " pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:33:11.060846 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:11.060702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-console-oauth-config\") pod \"console-5f7b644f79-shxdr\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") " pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:33:11.061462 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:11.061428    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-oauth-serving-cert\") pod \"console-5f7b644f79-shxdr\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") " pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:33:11.061667 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:11.061542    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-console-config\") pod \"console-5f7b644f79-shxdr\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") " pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:33:11.061730 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:11.061671    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-service-ca\") pod \"console-5f7b644f79-shxdr\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") " pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:33:11.061794 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:11.061771    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-trusted-ca-bundle\") pod \"console-5f7b644f79-shxdr\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") " pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:33:11.063622 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:11.063590    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-console-oauth-config\") pod \"console-5f7b644f79-shxdr\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") " pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:33:11.063763 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:11.063629    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-console-serving-cert\") pod \"console-5f7b644f79-shxdr\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") " pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:33:11.068685 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:11.068656    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2292b\" (UniqueName: \"kubernetes.io/projected/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-kube-api-access-2292b\") pod \"console-5f7b644f79-shxdr\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") " pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:33:11.212552 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:11.212452    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:33:11.353854 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:11.353820    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f7b644f79-shxdr"]
Jul 07 06:33:11.357393 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:33:11.357360    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd99afd9b_0c46_4c10_821c_b16b6b4bbd76.slice/crio-807502ca99bb5c7ba9b64571e789320a907353499a74d78edab40b10ff6d2222 WatchSource:0}: Error finding container 807502ca99bb5c7ba9b64571e789320a907353499a74d78edab40b10ff6d2222: Status 404 returned error can't find the container with id 807502ca99bb5c7ba9b64571e789320a907353499a74d78edab40b10ff6d2222
Jul 07 06:33:11.667093 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:11.667056    2563 generic.go:356] "Generic (PLEG): container finished" podID="5f6429ad-0f44-4eaa-9638-0627dc4e392a" containerID="ee68d089f0c525213fad4297c1586e53726d7231d6e1ad94fa56ad4bafb7b4f1" exitCode=0
Jul 07 06:33:11.667581 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:11.667145    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f6429ad-0f44-4eaa-9638-0627dc4e392a","Type":"ContainerDied","Data":"ee68d089f0c525213fad4297c1586e53726d7231d6e1ad94fa56ad4bafb7b4f1"}
Jul 07 06:33:11.668719 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:11.668692    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-p5kjp" event={"ID":"26aef36b-c5d2-4c16-9bfc-917f6eba2bb9","Type":"ContainerStarted","Data":"73094d860a599bc6067b46d1aedf3eb1636662cead2c925a9b736f4068896a90"}
Jul 07 06:33:11.668915 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:11.668888    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-p5kjp"
Jul 07 06:33:11.670598 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:11.670575    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f7b644f79-shxdr" event={"ID":"d99afd9b-0c46-4c10-821c-b16b6b4bbd76","Type":"ContainerStarted","Data":"5718d4859a360c443d6a00f79835357887b92d11ac9652395fa050ffcc7ae4c5"}
Jul 07 06:33:11.670700 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:11.670620    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f7b644f79-shxdr" event={"ID":"d99afd9b-0c46-4c10-821c-b16b6b4bbd76","Type":"ContainerStarted","Data":"807502ca99bb5c7ba9b64571e789320a907353499a74d78edab40b10ff6d2222"}
Jul 07 06:33:11.674153 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:11.674135    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-p5kjp"
Jul 07 06:33:11.764175 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:11.764120    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f7b644f79-shxdr" podStartSLOduration=1.764104443 podStartE2EDuration="1.764104443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:33:11.762905197 +0000 UTC m=+75.127101767" watchObservedRunningTime="2026-07-07 06:33:11.764104443 +0000 UTC m=+75.128300987"
Jul 07 06:33:11.779077 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:11.779019    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-p5kjp" podStartSLOduration=4.958481906 podStartE2EDuration="6.778999526s" totalImagesPullingTime="1.82051762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:33:11.77622541 +0000 UTC m=+75.140421972" watchObservedRunningTime="2026-07-07 06:33:11.778999526 +0000 UTC m=+75.143196076"
Jul 07 06:33:15.693241 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:15.693208    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f6429ad-0f44-4eaa-9638-0627dc4e392a","Type":"ContainerStarted","Data":"780d07d10fa53dfb4e43257b02760ef6c32c92a21afee08bc7c71b8bd3b1cf76"}
Jul 07 06:33:15.693698 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:15.693274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f6429ad-0f44-4eaa-9638-0627dc4e392a","Type":"ContainerStarted","Data":"e9d84eb19c990af029f67e9b31180c74297ba0c6a2c00e8fea82a36531555ecb"}
Jul 07 06:33:16.294883 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:16.294842    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:16.295070 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:16.294909    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:16.301708 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:16.301673    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:16.703119 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:16.703091    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:17.705808 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:17.705777    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f6429ad-0f44-4eaa-9638-0627dc4e392a","Type":"ContainerStarted","Data":"8e761c428777b7ef07f09d1e2f89a0ee5e5a75b7889d54b962b005abc5d1056d"}
Jul 07 06:33:17.706209 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:17.705842    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f6429ad-0f44-4eaa-9638-0627dc4e392a","Type":"ContainerStarted","Data":"dbe183b50bf39c7a812d731b80f1eb2f1e615fef55b0d21b5ea38415f8190365"}
Jul 07 06:33:17.706209 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:17.705872    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f6429ad-0f44-4eaa-9638-0627dc4e392a","Type":"ContainerStarted","Data":"97ed51859279c867531c419062969421c970c7994a6f13343fd20bceddb90bb7"}
Jul 07 06:33:17.706209 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:17.705898    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f6429ad-0f44-4eaa-9638-0627dc4e392a","Type":"ContainerStarted","Data":"7e7cf253fd51e0ab0856cdeb1788540b2953849585e00981040560b94c5e8249"}
Jul 07 06:33:17.739624 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:17.739560    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.640686911 podStartE2EDuration="10.739538816s" totalImagesPullingTime="7.072085173s" totalInitContainerRuntime="26.766732ms" isStatefulPod=false podCreationTimestamp="2026-07-07 06:33:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:33:17.737371228 +0000 UTC m=+81.101567767" watchObservedRunningTime="2026-07-07 06:33:17.739538816 +0000 UTC m=+81.103735362"
Jul 07 06:33:17.743705 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:17.743672    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:33:21.213385 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:21.213343    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:33:21.213908 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:21.213410    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:33:21.218490 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:21.218464    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:33:21.726018 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:21.725981    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:33:21.781135 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:21.781076    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84b69f5449-7d49k"]
Jul 07 06:33:25.847165 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:25.847127    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-f7646d95-nsbd6"
Jul 07 06:33:25.847662 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:25.847194    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-f7646d95-nsbd6"
Jul 07 06:33:30.983500 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:30.983446    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7f45b5f89f-t2pnz" podUID="b6427e28-9acc-4588-ac81-a9c40bdee78e" containerName="console" containerID="cri-o://8dc5be401f6efb45cf79869183277374510ed8c840115dbec5fec090acb3bf32" gracePeriod=15
Jul 07 06:33:31.234714 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.234629    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f45b5f89f-t2pnz"
Jul 07 06:33:31.345111 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.345072    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qtpk\" (UniqueName: \"kubernetes.io/projected/b6427e28-9acc-4588-ac81-a9c40bdee78e-kube-api-access-9qtpk\") pod \"b6427e28-9acc-4588-ac81-a9c40bdee78e\" (UID: \"b6427e28-9acc-4588-ac81-a9c40bdee78e\") "
Jul 07 06:33:31.345111 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.345116    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6427e28-9acc-4588-ac81-a9c40bdee78e-service-ca\") pod \"b6427e28-9acc-4588-ac81-a9c40bdee78e\" (UID: \"b6427e28-9acc-4588-ac81-a9c40bdee78e\") "
Jul 07 06:33:31.345345 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.345147    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6427e28-9acc-4588-ac81-a9c40bdee78e-oauth-serving-cert\") pod \"b6427e28-9acc-4588-ac81-a9c40bdee78e\" (UID: \"b6427e28-9acc-4588-ac81-a9c40bdee78e\") "
Jul 07 06:33:31.345345 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.345173    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6427e28-9acc-4588-ac81-a9c40bdee78e-console-serving-cert\") pod \"b6427e28-9acc-4588-ac81-a9c40bdee78e\" (UID: \"b6427e28-9acc-4588-ac81-a9c40bdee78e\") "
Jul 07 06:33:31.345345 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.345225    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6427e28-9acc-4588-ac81-a9c40bdee78e-console-config\") pod \"b6427e28-9acc-4588-ac81-a9c40bdee78e\" (UID: \"b6427e28-9acc-4588-ac81-a9c40bdee78e\") "
Jul 07 06:33:31.345345 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.345268    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6427e28-9acc-4588-ac81-a9c40bdee78e-console-oauth-config\") pod \"b6427e28-9acc-4588-ac81-a9c40bdee78e\" (UID: \"b6427e28-9acc-4588-ac81-a9c40bdee78e\") "
Jul 07 06:33:31.345597 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.345567    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6427e28-9acc-4588-ac81-a9c40bdee78e-service-ca" (OuterVolumeSpecName: "service-ca") pod "b6427e28-9acc-4588-ac81-a9c40bdee78e" (UID: "b6427e28-9acc-4588-ac81-a9c40bdee78e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 06:33:31.345692 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.345664    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6427e28-9acc-4588-ac81-a9c40bdee78e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b6427e28-9acc-4588-ac81-a9c40bdee78e" (UID: "b6427e28-9acc-4588-ac81-a9c40bdee78e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 06:33:31.345744 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.345680    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6427e28-9acc-4588-ac81-a9c40bdee78e-console-config" (OuterVolumeSpecName: "console-config") pod "b6427e28-9acc-4588-ac81-a9c40bdee78e" (UID: "b6427e28-9acc-4588-ac81-a9c40bdee78e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 06:33:31.347834 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.347806    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6427e28-9acc-4588-ac81-a9c40bdee78e-kube-api-access-9qtpk" (OuterVolumeSpecName: "kube-api-access-9qtpk") pod "b6427e28-9acc-4588-ac81-a9c40bdee78e" (UID: "b6427e28-9acc-4588-ac81-a9c40bdee78e"). InnerVolumeSpecName "kube-api-access-9qtpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 06:33:31.347944 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.347826    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6427e28-9acc-4588-ac81-a9c40bdee78e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b6427e28-9acc-4588-ac81-a9c40bdee78e" (UID: "b6427e28-9acc-4588-ac81-a9c40bdee78e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 06:33:31.347944 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.347871    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6427e28-9acc-4588-ac81-a9c40bdee78e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b6427e28-9acc-4588-ac81-a9c40bdee78e" (UID: "b6427e28-9acc-4588-ac81-a9c40bdee78e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 06:33:31.445919 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.445878    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qtpk\" (UniqueName: \"kubernetes.io/projected/b6427e28-9acc-4588-ac81-a9c40bdee78e-kube-api-access-9qtpk\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:33:31.445919 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.445911    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6427e28-9acc-4588-ac81-a9c40bdee78e-service-ca\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:33:31.445919 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.445925    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6427e28-9acc-4588-ac81-a9c40bdee78e-oauth-serving-cert\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:33:31.446168 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.445939    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6427e28-9acc-4588-ac81-a9c40bdee78e-console-serving-cert\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:33:31.446168 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.445950    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6427e28-9acc-4588-ac81-a9c40bdee78e-console-config\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:33:31.446168 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.445961    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6427e28-9acc-4588-ac81-a9c40bdee78e-console-oauth-config\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:33:31.766985 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.766945    2563 generic.go:356] "Generic (PLEG): container finished" podID="b6427e28-9acc-4588-ac81-a9c40bdee78e" containerID="8dc5be401f6efb45cf79869183277374510ed8c840115dbec5fec090acb3bf32" exitCode=0
Jul 07 06:33:31.767157 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.767013    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f45b5f89f-t2pnz"
Jul 07 06:33:31.767157 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.767035    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f45b5f89f-t2pnz" event={"ID":"b6427e28-9acc-4588-ac81-a9c40bdee78e","Type":"ContainerDied","Data":"8dc5be401f6efb45cf79869183277374510ed8c840115dbec5fec090acb3bf32"}
Jul 07 06:33:31.767157 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.767128    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f45b5f89f-t2pnz" event={"ID":"b6427e28-9acc-4588-ac81-a9c40bdee78e","Type":"ContainerDied","Data":"3ab9a1df96835eb00a45a7f8c866e8e4de8d72587302e7c267b68203bf78a962"}
Jul 07 06:33:31.767286 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.767158    2563 scope.go:117] "RemoveContainer" containerID="8dc5be401f6efb45cf79869183277374510ed8c840115dbec5fec090acb3bf32"
Jul 07 06:33:31.775994 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.775979    2563 scope.go:117] "RemoveContainer" containerID="8dc5be401f6efb45cf79869183277374510ed8c840115dbec5fec090acb3bf32"
Jul 07 06:33:31.776260 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:33:31.776240    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc5be401f6efb45cf79869183277374510ed8c840115dbec5fec090acb3bf32\": container with ID starting with 8dc5be401f6efb45cf79869183277374510ed8c840115dbec5fec090acb3bf32 not found: ID does not exist" containerID="8dc5be401f6efb45cf79869183277374510ed8c840115dbec5fec090acb3bf32"
Jul 07 06:33:31.776310 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.776269    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc5be401f6efb45cf79869183277374510ed8c840115dbec5fec090acb3bf32"} err="failed to get container status \"8dc5be401f6efb45cf79869183277374510ed8c840115dbec5fec090acb3bf32\": rpc error: code = NotFound desc = could not find container \"8dc5be401f6efb45cf79869183277374510ed8c840115dbec5fec090acb3bf32\": container with ID starting with 8dc5be401f6efb45cf79869183277374510ed8c840115dbec5fec090acb3bf32 not found: ID does not exist"
Jul 07 06:33:31.787671 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.787627    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f45b5f89f-t2pnz"]
Jul 07 06:33:31.791189 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:31.791166    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f45b5f89f-t2pnz"]
Jul 07 06:33:33.171990 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:33.171957    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6427e28-9acc-4588-ac81-a9c40bdee78e" path="/var/lib/kubelet/pods/b6427e28-9acc-4588-ac81-a9c40bdee78e/volumes"
Jul 07 06:33:35.796182 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:35.796110    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5886655f8b-qm2fn" podUID="feddeb52-1647-49c2-98ab-7cfdfade2460" containerName="console" containerID="cri-o://999db51b821d5155fe47b19bced361a85e217db1fe1ac209cdd21e91a00c3e93" gracePeriod=15
Jul 07 06:33:36.049092 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.049026    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5886655f8b-qm2fn"
Jul 07 06:33:36.084682 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.084620    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/feddeb52-1647-49c2-98ab-7cfdfade2460-console-serving-cert\") pod \"feddeb52-1647-49c2-98ab-7cfdfade2460\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") "
Jul 07 06:33:36.084682 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.084686    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fg59f\" (UniqueName: \"kubernetes.io/projected/feddeb52-1647-49c2-98ab-7cfdfade2460-kube-api-access-fg59f\") pod \"feddeb52-1647-49c2-98ab-7cfdfade2460\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") "
Jul 07 06:33:36.084923 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.084712    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feddeb52-1647-49c2-98ab-7cfdfade2460-trusted-ca-bundle\") pod \"feddeb52-1647-49c2-98ab-7cfdfade2460\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") "
Jul 07 06:33:36.084923 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.084748    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/feddeb52-1647-49c2-98ab-7cfdfade2460-service-ca\") pod \"feddeb52-1647-49c2-98ab-7cfdfade2460\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") "
Jul 07 06:33:36.084923 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.084810    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/feddeb52-1647-49c2-98ab-7cfdfade2460-console-oauth-config\") pod \"feddeb52-1647-49c2-98ab-7cfdfade2460\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") "
Jul 07 06:33:36.084923 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.084853    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/feddeb52-1647-49c2-98ab-7cfdfade2460-oauth-serving-cert\") pod \"feddeb52-1647-49c2-98ab-7cfdfade2460\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") "
Jul 07 06:33:36.084923 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.084884    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/feddeb52-1647-49c2-98ab-7cfdfade2460-console-config\") pod \"feddeb52-1647-49c2-98ab-7cfdfade2460\" (UID: \"feddeb52-1647-49c2-98ab-7cfdfade2460\") "
Jul 07 06:33:36.085249 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.085222    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feddeb52-1647-49c2-98ab-7cfdfade2460-service-ca" (OuterVolumeSpecName: "service-ca") pod "feddeb52-1647-49c2-98ab-7cfdfade2460" (UID: "feddeb52-1647-49c2-98ab-7cfdfade2460"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 06:33:36.085355 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.085267    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feddeb52-1647-49c2-98ab-7cfdfade2460-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "feddeb52-1647-49c2-98ab-7cfdfade2460" (UID: "feddeb52-1647-49c2-98ab-7cfdfade2460"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 06:33:36.085412 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.085387    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feddeb52-1647-49c2-98ab-7cfdfade2460-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "feddeb52-1647-49c2-98ab-7cfdfade2460" (UID: "feddeb52-1647-49c2-98ab-7cfdfade2460"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 06:33:36.085679 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.085623    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feddeb52-1647-49c2-98ab-7cfdfade2460-console-config" (OuterVolumeSpecName: "console-config") pod "feddeb52-1647-49c2-98ab-7cfdfade2460" (UID: "feddeb52-1647-49c2-98ab-7cfdfade2460"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 06:33:36.087209 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.087178    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feddeb52-1647-49c2-98ab-7cfdfade2460-kube-api-access-fg59f" (OuterVolumeSpecName: "kube-api-access-fg59f") pod "feddeb52-1647-49c2-98ab-7cfdfade2460" (UID: "feddeb52-1647-49c2-98ab-7cfdfade2460"). InnerVolumeSpecName "kube-api-access-fg59f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 06:33:36.087209 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.087180    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feddeb52-1647-49c2-98ab-7cfdfade2460-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "feddeb52-1647-49c2-98ab-7cfdfade2460" (UID: "feddeb52-1647-49c2-98ab-7cfdfade2460"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 06:33:36.087554 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.087534    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feddeb52-1647-49c2-98ab-7cfdfade2460-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "feddeb52-1647-49c2-98ab-7cfdfade2460" (UID: "feddeb52-1647-49c2-98ab-7cfdfade2460"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 06:33:36.186447 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.186397    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg59f\" (UniqueName: \"kubernetes.io/projected/feddeb52-1647-49c2-98ab-7cfdfade2460-kube-api-access-fg59f\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:33:36.186447 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.186435    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/feddeb52-1647-49c2-98ab-7cfdfade2460-trusted-ca-bundle\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:33:36.186447 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.186449    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/feddeb52-1647-49c2-98ab-7cfdfade2460-service-ca\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:33:36.186736 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.186461    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/feddeb52-1647-49c2-98ab-7cfdfade2460-console-oauth-config\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:33:36.186736 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.186473    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/feddeb52-1647-49c2-98ab-7cfdfade2460-oauth-serving-cert\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:33:36.186736 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.186484    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/feddeb52-1647-49c2-98ab-7cfdfade2460-console-config\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:33:36.186736 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.186495    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/feddeb52-1647-49c2-98ab-7cfdfade2460-console-serving-cert\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:33:36.787874 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.787837    2563 generic.go:356] "Generic (PLEG): container finished" podID="feddeb52-1647-49c2-98ab-7cfdfade2460" containerID="999db51b821d5155fe47b19bced361a85e217db1fe1ac209cdd21e91a00c3e93" exitCode=0
Jul 07 06:33:36.788124 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.787884    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5886655f8b-qm2fn"
Jul 07 06:33:36.788124 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.787912    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5886655f8b-qm2fn" event={"ID":"feddeb52-1647-49c2-98ab-7cfdfade2460","Type":"ContainerDied","Data":"999db51b821d5155fe47b19bced361a85e217db1fe1ac209cdd21e91a00c3e93"}
Jul 07 06:33:36.788124 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.787959    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5886655f8b-qm2fn" event={"ID":"feddeb52-1647-49c2-98ab-7cfdfade2460","Type":"ContainerDied","Data":"d018dbf9919e25a4f590db314a6463cadbf755eadb42fff694870e829149a9f6"}
Jul 07 06:33:36.788124 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.787981    2563 scope.go:117] "RemoveContainer" containerID="999db51b821d5155fe47b19bced361a85e217db1fe1ac209cdd21e91a00c3e93"
Jul 07 06:33:36.796494 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.796475    2563 scope.go:117] "RemoveContainer" containerID="999db51b821d5155fe47b19bced361a85e217db1fe1ac209cdd21e91a00c3e93"
Jul 07 06:33:36.796783 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:33:36.796766    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"999db51b821d5155fe47b19bced361a85e217db1fe1ac209cdd21e91a00c3e93\": container with ID starting with 999db51b821d5155fe47b19bced361a85e217db1fe1ac209cdd21e91a00c3e93 not found: ID does not exist" containerID="999db51b821d5155fe47b19bced361a85e217db1fe1ac209cdd21e91a00c3e93"
Jul 07 06:33:36.796824 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.796794    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"999db51b821d5155fe47b19bced361a85e217db1fe1ac209cdd21e91a00c3e93"} err="failed to get container status \"999db51b821d5155fe47b19bced361a85e217db1fe1ac209cdd21e91a00c3e93\": rpc error: code = NotFound desc = could not find container \"999db51b821d5155fe47b19bced361a85e217db1fe1ac209cdd21e91a00c3e93\": container with ID starting with 999db51b821d5155fe47b19bced361a85e217db1fe1ac209cdd21e91a00c3e93 not found: ID does not exist"
Jul 07 06:33:36.808073 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.808049    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5886655f8b-qm2fn"]
Jul 07 06:33:36.812011 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:36.811990    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5886655f8b-qm2fn"]
Jul 07 06:33:37.171141 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:37.171114    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="feddeb52-1647-49c2-98ab-7cfdfade2460" path="/var/lib/kubelet/pods/feddeb52-1647-49c2-98ab-7cfdfade2460/volumes"
Jul 07 06:33:40.663548 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:40.663516    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jlprw"
Jul 07 06:33:45.853586 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:45.853550    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-f7646d95-nsbd6"
Jul 07 06:33:45.857733 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:45.857714    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-f7646d95-nsbd6"
Jul 07 06:33:46.810988 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:46.810930    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-84b69f5449-7d49k" podUID="3eec4752-565d-4551-b768-371ad13fdc5b" containerName="console" containerID="cri-o://bf7cbd6fe2b15941a3ee417d4f6c6ece197e5a711bbc8aaabac9d5836e5cbd8c" gracePeriod=15
Jul 07 06:33:47.081201 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.081142    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:47.186215 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.186181    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3eec4752-565d-4551-b768-371ad13fdc5b-console-serving-cert\") pod \"3eec4752-565d-4551-b768-371ad13fdc5b\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") "
Jul 07 06:33:47.186215 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.186219    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3eec4752-565d-4551-b768-371ad13fdc5b-oauth-serving-cert\") pod \"3eec4752-565d-4551-b768-371ad13fdc5b\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") "
Jul 07 06:33:47.186404 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.186239    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4v9tn\" (UniqueName: \"kubernetes.io/projected/3eec4752-565d-4551-b768-371ad13fdc5b-kube-api-access-4v9tn\") pod \"3eec4752-565d-4551-b768-371ad13fdc5b\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") "
Jul 07 06:33:47.186404 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.186278    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eec4752-565d-4551-b768-371ad13fdc5b-trusted-ca-bundle\") pod \"3eec4752-565d-4551-b768-371ad13fdc5b\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") "
Jul 07 06:33:47.186404 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.186316    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3eec4752-565d-4551-b768-371ad13fdc5b-console-oauth-config\") pod \"3eec4752-565d-4551-b768-371ad13fdc5b\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") "
Jul 07 06:33:47.186404 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.186351    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3eec4752-565d-4551-b768-371ad13fdc5b-console-config\") pod \"3eec4752-565d-4551-b768-371ad13fdc5b\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") "
Jul 07 06:33:47.186404 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.186383    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3eec4752-565d-4551-b768-371ad13fdc5b-service-ca\") pod \"3eec4752-565d-4551-b768-371ad13fdc5b\" (UID: \"3eec4752-565d-4551-b768-371ad13fdc5b\") "
Jul 07 06:33:47.186838 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.186809    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eec4752-565d-4551-b768-371ad13fdc5b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3eec4752-565d-4551-b768-371ad13fdc5b" (UID: "3eec4752-565d-4551-b768-371ad13fdc5b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 06:33:47.186838 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.186819    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eec4752-565d-4551-b768-371ad13fdc5b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3eec4752-565d-4551-b768-371ad13fdc5b" (UID: "3eec4752-565d-4551-b768-371ad13fdc5b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 06:33:47.186838 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.186823    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eec4752-565d-4551-b768-371ad13fdc5b-console-config" (OuterVolumeSpecName: "console-config") pod "3eec4752-565d-4551-b768-371ad13fdc5b" (UID: "3eec4752-565d-4551-b768-371ad13fdc5b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 06:33:47.187218 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.186918    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eec4752-565d-4551-b768-371ad13fdc5b-service-ca" (OuterVolumeSpecName: "service-ca") pod "3eec4752-565d-4551-b768-371ad13fdc5b" (UID: "3eec4752-565d-4551-b768-371ad13fdc5b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 06:33:47.188647 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.188610    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eec4752-565d-4551-b768-371ad13fdc5b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3eec4752-565d-4551-b768-371ad13fdc5b" (UID: "3eec4752-565d-4551-b768-371ad13fdc5b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 06:33:47.188647 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.188611    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eec4752-565d-4551-b768-371ad13fdc5b-kube-api-access-4v9tn" (OuterVolumeSpecName: "kube-api-access-4v9tn") pod "3eec4752-565d-4551-b768-371ad13fdc5b" (UID: "3eec4752-565d-4551-b768-371ad13fdc5b"). InnerVolumeSpecName "kube-api-access-4v9tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 06:33:47.188773 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.188694    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eec4752-565d-4551-b768-371ad13fdc5b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3eec4752-565d-4551-b768-371ad13fdc5b" (UID: "3eec4752-565d-4551-b768-371ad13fdc5b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 06:33:47.287772 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.287737    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3eec4752-565d-4551-b768-371ad13fdc5b-console-oauth-config\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:33:47.287772 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.287769    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3eec4752-565d-4551-b768-371ad13fdc5b-console-config\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:33:47.287772 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.287779    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3eec4752-565d-4551-b768-371ad13fdc5b-service-ca\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:33:47.288016 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.287787    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3eec4752-565d-4551-b768-371ad13fdc5b-console-serving-cert\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:33:47.288016 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.287797    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3eec4752-565d-4551-b768-371ad13fdc5b-oauth-serving-cert\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:33:47.288016 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.287807    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v9tn\" (UniqueName: \"kubernetes.io/projected/3eec4752-565d-4551-b768-371ad13fdc5b-kube-api-access-4v9tn\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:33:47.288016 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.287815    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eec4752-565d-4551-b768-371ad13fdc5b-trusted-ca-bundle\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:33:47.833360 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.833324    2563 generic.go:356] "Generic (PLEG): container finished" podID="3eec4752-565d-4551-b768-371ad13fdc5b" containerID="bf7cbd6fe2b15941a3ee417d4f6c6ece197e5a711bbc8aaabac9d5836e5cbd8c" exitCode=0
Jul 07 06:33:47.833567 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.833381    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b69f5449-7d49k"
Jul 07 06:33:47.833567 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.833407    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b69f5449-7d49k" event={"ID":"3eec4752-565d-4551-b768-371ad13fdc5b","Type":"ContainerDied","Data":"bf7cbd6fe2b15941a3ee417d4f6c6ece197e5a711bbc8aaabac9d5836e5cbd8c"}
Jul 07 06:33:47.833567 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.833490    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b69f5449-7d49k" event={"ID":"3eec4752-565d-4551-b768-371ad13fdc5b","Type":"ContainerDied","Data":"7ae1de435f07fc6a0199e6701a01385c88e25ce3c83cd9d530300c17538f2bda"}
Jul 07 06:33:47.833567 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.833518    2563 scope.go:117] "RemoveContainer" containerID="bf7cbd6fe2b15941a3ee417d4f6c6ece197e5a711bbc8aaabac9d5836e5cbd8c"
Jul 07 06:33:47.842373 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.842354    2563 scope.go:117] "RemoveContainer" containerID="bf7cbd6fe2b15941a3ee417d4f6c6ece197e5a711bbc8aaabac9d5836e5cbd8c"
Jul 07 06:33:47.842668 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:33:47.842624    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf7cbd6fe2b15941a3ee417d4f6c6ece197e5a711bbc8aaabac9d5836e5cbd8c\": container with ID starting with bf7cbd6fe2b15941a3ee417d4f6c6ece197e5a711bbc8aaabac9d5836e5cbd8c not found: ID does not exist" containerID="bf7cbd6fe2b15941a3ee417d4f6c6ece197e5a711bbc8aaabac9d5836e5cbd8c"
Jul 07 06:33:47.842727 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.842680    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf7cbd6fe2b15941a3ee417d4f6c6ece197e5a711bbc8aaabac9d5836e5cbd8c"} err="failed to get container status \"bf7cbd6fe2b15941a3ee417d4f6c6ece197e5a711bbc8aaabac9d5836e5cbd8c\": rpc error: code = NotFound desc = could not find container \"bf7cbd6fe2b15941a3ee417d4f6c6ece197e5a711bbc8aaabac9d5836e5cbd8c\": container with ID starting with bf7cbd6fe2b15941a3ee417d4f6c6ece197e5a711bbc8aaabac9d5836e5cbd8c not found: ID does not exist"
Jul 07 06:33:47.854371 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.854341    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84b69f5449-7d49k"]
Jul 07 06:33:47.857621 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:47.857598    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84b69f5449-7d49k"]
Jul 07 06:33:49.171431 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:33:49.171394    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3eec4752-565d-4551-b768-371ad13fdc5b" path="/var/lib/kubelet/pods/3eec4752-565d-4551-b768-371ad13fdc5b/volumes"
Jul 07 06:34:07.743496 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:07.743461    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:34:07.764024 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:07.763991    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:34:07.927619 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:07.927593    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 06:34:20.361369 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.361337    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-799b758f7c-k7jp8"]
Jul 07 06:34:20.361866 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.361756    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feddeb52-1647-49c2-98ab-7cfdfade2460" containerName="console"
Jul 07 06:34:20.361866 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.361774    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="feddeb52-1647-49c2-98ab-7cfdfade2460" containerName="console"
Jul 07 06:34:20.361866 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.361792    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6427e28-9acc-4588-ac81-a9c40bdee78e" containerName="console"
Jul 07 06:34:20.361866 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.361800    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6427e28-9acc-4588-ac81-a9c40bdee78e" containerName="console"
Jul 07 06:34:20.361866 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.361810    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eec4752-565d-4551-b768-371ad13fdc5b" containerName="console"
Jul 07 06:34:20.361866 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.361818    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eec4752-565d-4551-b768-371ad13fdc5b" containerName="console"
Jul 07 06:34:20.362161 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.361897    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6427e28-9acc-4588-ac81-a9c40bdee78e" containerName="console"
Jul 07 06:34:20.362161 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.361910    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eec4752-565d-4551-b768-371ad13fdc5b" containerName="console"
Jul 07 06:34:20.362161 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.361919    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="feddeb52-1647-49c2-98ab-7cfdfade2460" containerName="console"
Jul 07 06:34:20.363977 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.363959    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:34:20.375147 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.375127    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-799b758f7c-k7jp8"]
Jul 07 06:34:20.473761 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.473730    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/009a488d-d184-4adc-a788-7cfc4991d906-console-serving-cert\") pod \"console-799b758f7c-k7jp8\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") " pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:34:20.473761 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.473763    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/009a488d-d184-4adc-a788-7cfc4991d906-service-ca\") pod \"console-799b758f7c-k7jp8\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") " pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:34:20.473940 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.473789    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/009a488d-d184-4adc-a788-7cfc4991d906-console-oauth-config\") pod \"console-799b758f7c-k7jp8\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") " pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:34:20.473940 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.473828    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/009a488d-d184-4adc-a788-7cfc4991d906-trusted-ca-bundle\") pod \"console-799b758f7c-k7jp8\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") " pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:34:20.473940 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.473907    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/009a488d-d184-4adc-a788-7cfc4991d906-console-config\") pod \"console-799b758f7c-k7jp8\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") " pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:34:20.474031 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.474008    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb4bz\" (UniqueName: \"kubernetes.io/projected/009a488d-d184-4adc-a788-7cfc4991d906-kube-api-access-gb4bz\") pod \"console-799b758f7c-k7jp8\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") " pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:34:20.474031 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.474028    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/009a488d-d184-4adc-a788-7cfc4991d906-oauth-serving-cert\") pod \"console-799b758f7c-k7jp8\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") " pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:34:20.574844 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.574816    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb4bz\" (UniqueName: \"kubernetes.io/projected/009a488d-d184-4adc-a788-7cfc4991d906-kube-api-access-gb4bz\") pod \"console-799b758f7c-k7jp8\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") " pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:34:20.574844 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.574847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/009a488d-d184-4adc-a788-7cfc4991d906-oauth-serving-cert\") pod \"console-799b758f7c-k7jp8\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") " pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:34:20.575014 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.574877    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/009a488d-d184-4adc-a788-7cfc4991d906-console-serving-cert\") pod \"console-799b758f7c-k7jp8\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") " pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:34:20.575014 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.574899    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/009a488d-d184-4adc-a788-7cfc4991d906-service-ca\") pod \"console-799b758f7c-k7jp8\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") " pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:34:20.575014 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.574931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/009a488d-d184-4adc-a788-7cfc4991d906-console-oauth-config\") pod \"console-799b758f7c-k7jp8\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") " pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:34:20.575014 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.574956    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/009a488d-d184-4adc-a788-7cfc4991d906-trusted-ca-bundle\") pod \"console-799b758f7c-k7jp8\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") " pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:34:20.575014 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.575001    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/009a488d-d184-4adc-a788-7cfc4991d906-console-config\") pod \"console-799b758f7c-k7jp8\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") " pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:34:20.575612 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.575588    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/009a488d-d184-4adc-a788-7cfc4991d906-service-ca\") pod \"console-799b758f7c-k7jp8\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") " pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:34:20.575741 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.575610    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/009a488d-d184-4adc-a788-7cfc4991d906-oauth-serving-cert\") pod \"console-799b758f7c-k7jp8\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") " pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:34:20.575786 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.575740    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/009a488d-d184-4adc-a788-7cfc4991d906-console-config\") pod \"console-799b758f7c-k7jp8\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") " pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:34:20.575907 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.575889    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/009a488d-d184-4adc-a788-7cfc4991d906-trusted-ca-bundle\") pod \"console-799b758f7c-k7jp8\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") " pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:34:20.578036 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.578006    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/009a488d-d184-4adc-a788-7cfc4991d906-console-oauth-config\") pod \"console-799b758f7c-k7jp8\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") " pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:34:20.578238 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.578216    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/009a488d-d184-4adc-a788-7cfc4991d906-console-serving-cert\") pod \"console-799b758f7c-k7jp8\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") " pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:34:20.585532 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.585509    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb4bz\" (UniqueName: \"kubernetes.io/projected/009a488d-d184-4adc-a788-7cfc4991d906-kube-api-access-gb4bz\") pod \"console-799b758f7c-k7jp8\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") " pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:34:20.672839 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.672788    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:34:20.796968 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.796937    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-799b758f7c-k7jp8"]
Jul 07 06:34:20.800294 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:34:20.800268    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod009a488d_d184_4adc_a788_7cfc4991d906.slice/crio-016b48508c97425e858e500059fdd1e9026e9402f744521fa8a8c8558005f374 WatchSource:0}: Error finding container 016b48508c97425e858e500059fdd1e9026e9402f744521fa8a8c8558005f374: Status 404 returned error can't find the container with id 016b48508c97425e858e500059fdd1e9026e9402f744521fa8a8c8558005f374
Jul 07 06:34:20.964424 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.964349    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-799b758f7c-k7jp8" event={"ID":"009a488d-d184-4adc-a788-7cfc4991d906","Type":"ContainerStarted","Data":"c25ccaabb7f2eb53fbe052834d82ced611d9d8f047c600655e4ccc17b035d0f3"}
Jul 07 06:34:20.964424 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.964393    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-799b758f7c-k7jp8" event={"ID":"009a488d-d184-4adc-a788-7cfc4991d906","Type":"ContainerStarted","Data":"016b48508c97425e858e500059fdd1e9026e9402f744521fa8a8c8558005f374"}
Jul 07 06:34:20.982034 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:20.981993    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-799b758f7c-k7jp8" podStartSLOduration=0.981975502 podStartE2EDuration="981.975502ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:34:20.980557033 +0000 UTC m=+144.344753594" watchObservedRunningTime="2026-07-07 06:34:20.981975502 +0000 UTC m=+144.346172112"
Jul 07 06:34:30.673254 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:30.673213    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:34:30.673824 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:30.673272    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:34:30.678235 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:30.678209    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:34:31.006402 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:31.006322    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:34:31.056972 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:31.056941    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f7b644f79-shxdr"]
Jul 07 06:34:56.078497 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:56.078443    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5f7b644f79-shxdr" podUID="d99afd9b-0c46-4c10-821c-b16b6b4bbd76" containerName="console" containerID="cri-o://5718d4859a360c443d6a00f79835357887b92d11ac9652395fa050ffcc7ae4c5" gracePeriod=15
Jul 07 06:34:56.322269 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:56.322249    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:34:56.369353 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:56.369289    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-service-ca\") pod \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") "
Jul 07 06:34:56.369353 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:56.369333    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-console-serving-cert\") pod \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") "
Jul 07 06:34:56.369523 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:56.369355    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-oauth-serving-cert\") pod \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") "
Jul 07 06:34:56.369523 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:56.369385    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-trusted-ca-bundle\") pod \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") "
Jul 07 06:34:56.369523 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:56.369402    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-console-config\") pod \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") "
Jul 07 06:34:56.369523 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:56.369440    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2292b\" (UniqueName: \"kubernetes.io/projected/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-kube-api-access-2292b\") pod \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") "
Jul 07 06:34:56.369523 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:56.369471    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-console-oauth-config\") pod \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\" (UID: \"d99afd9b-0c46-4c10-821c-b16b6b4bbd76\") "
Jul 07 06:34:56.369841 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:56.369812    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-service-ca" (OuterVolumeSpecName: "service-ca") pod "d99afd9b-0c46-4c10-821c-b16b6b4bbd76" (UID: "d99afd9b-0c46-4c10-821c-b16b6b4bbd76"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 06:34:56.369901 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:56.369841    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d99afd9b-0c46-4c10-821c-b16b6b4bbd76" (UID: "d99afd9b-0c46-4c10-821c-b16b6b4bbd76"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 06:34:56.370057 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:56.369965    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d99afd9b-0c46-4c10-821c-b16b6b4bbd76" (UID: "d99afd9b-0c46-4c10-821c-b16b6b4bbd76"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 06:34:56.370498 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:56.370473    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-console-config" (OuterVolumeSpecName: "console-config") pod "d99afd9b-0c46-4c10-821c-b16b6b4bbd76" (UID: "d99afd9b-0c46-4c10-821c-b16b6b4bbd76"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 06:34:56.371751 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:56.371718    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-kube-api-access-2292b" (OuterVolumeSpecName: "kube-api-access-2292b") pod "d99afd9b-0c46-4c10-821c-b16b6b4bbd76" (UID: "d99afd9b-0c46-4c10-821c-b16b6b4bbd76"). InnerVolumeSpecName "kube-api-access-2292b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 06:34:56.371836 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:56.371777    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d99afd9b-0c46-4c10-821c-b16b6b4bbd76" (UID: "d99afd9b-0c46-4c10-821c-b16b6b4bbd76"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 06:34:56.371836 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:56.371800    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d99afd9b-0c46-4c10-821c-b16b6b4bbd76" (UID: "d99afd9b-0c46-4c10-821c-b16b6b4bbd76"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 06:34:56.470952 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:56.470929    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-oauth-serving-cert\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:34:56.470952 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:56.470949    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-trusted-ca-bundle\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:34:56.471068 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:56.470958    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-console-config\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:34:56.471068 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:56.470967    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2292b\" (UniqueName: \"kubernetes.io/projected/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-kube-api-access-2292b\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:34:56.471068 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:56.470976    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-console-oauth-config\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:34:56.471068 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:56.470984    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-service-ca\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:34:56.471068 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:56.470991    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d99afd9b-0c46-4c10-821c-b16b6b4bbd76-console-serving-cert\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:34:57.094410 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:57.094378    2563 generic.go:356] "Generic (PLEG): container finished" podID="d99afd9b-0c46-4c10-821c-b16b6b4bbd76" containerID="5718d4859a360c443d6a00f79835357887b92d11ac9652395fa050ffcc7ae4c5" exitCode=0
Jul 07 06:34:57.094800 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:57.094434    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f7b644f79-shxdr"
Jul 07 06:34:57.094800 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:57.094441    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f7b644f79-shxdr" event={"ID":"d99afd9b-0c46-4c10-821c-b16b6b4bbd76","Type":"ContainerDied","Data":"5718d4859a360c443d6a00f79835357887b92d11ac9652395fa050ffcc7ae4c5"}
Jul 07 06:34:57.094800 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:57.094508    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f7b644f79-shxdr" event={"ID":"d99afd9b-0c46-4c10-821c-b16b6b4bbd76","Type":"ContainerDied","Data":"807502ca99bb5c7ba9b64571e789320a907353499a74d78edab40b10ff6d2222"}
Jul 07 06:34:57.094800 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:57.094528    2563 scope.go:117] "RemoveContainer" containerID="5718d4859a360c443d6a00f79835357887b92d11ac9652395fa050ffcc7ae4c5"
Jul 07 06:34:57.102700 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:57.102681    2563 scope.go:117] "RemoveContainer" containerID="5718d4859a360c443d6a00f79835357887b92d11ac9652395fa050ffcc7ae4c5"
Jul 07 06:34:57.102941 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:34:57.102924    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5718d4859a360c443d6a00f79835357887b92d11ac9652395fa050ffcc7ae4c5\": container with ID starting with 5718d4859a360c443d6a00f79835357887b92d11ac9652395fa050ffcc7ae4c5 not found: ID does not exist" containerID="5718d4859a360c443d6a00f79835357887b92d11ac9652395fa050ffcc7ae4c5"
Jul 07 06:34:57.102987 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:57.102949    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5718d4859a360c443d6a00f79835357887b92d11ac9652395fa050ffcc7ae4c5"} err="failed to get container status \"5718d4859a360c443d6a00f79835357887b92d11ac9652395fa050ffcc7ae4c5\": rpc error: code = NotFound desc = could not find container \"5718d4859a360c443d6a00f79835357887b92d11ac9652395fa050ffcc7ae4c5\": container with ID starting with 5718d4859a360c443d6a00f79835357887b92d11ac9652395fa050ffcc7ae4c5 not found: ID does not exist"
Jul 07 06:34:57.116239 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:57.116194    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f7b644f79-shxdr"]
Jul 07 06:34:57.119774 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:57.119757    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f7b644f79-shxdr"]
Jul 07 06:34:57.170697 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:34:57.170674    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d99afd9b-0c46-4c10-821c-b16b6b4bbd76" path="/var/lib/kubelet/pods/d99afd9b-0c46-4c10-821c-b16b6b4bbd76/volumes"
Jul 07 06:35:58.560783 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:35:58.560748    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-gk7fr"]
Jul 07 06:35:58.561240 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:35:58.561067    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d99afd9b-0c46-4c10-821c-b16b6b4bbd76" containerName="console"
Jul 07 06:35:58.561240 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:35:58.561081    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d99afd9b-0c46-4c10-821c-b16b6b4bbd76" containerName="console"
Jul 07 06:35:58.561240 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:35:58.561134    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d99afd9b-0c46-4c10-821c-b16b6b4bbd76" containerName="console"
Jul 07 06:35:58.563057 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:35:58.563042    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-gk7fr"
Jul 07 06:35:58.565050 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:35:58.565024    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 07 06:35:58.565150 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:35:58.565132    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 07 06:35:58.565556 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:35:58.565536    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-qzkj5\""
Jul 07 06:35:58.570102 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:35:58.570080    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbdx7\" (UniqueName: \"kubernetes.io/projected/033e0b80-fcc8-475a-9c22-d02216700d96-kube-api-access-qbdx7\") pod \"cert-manager-55b4d954d5-gk7fr\" (UID: \"033e0b80-fcc8-475a-9c22-d02216700d96\") " pod="cert-manager/cert-manager-55b4d954d5-gk7fr"
Jul 07 06:35:58.570198 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:35:58.570182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/033e0b80-fcc8-475a-9c22-d02216700d96-bound-sa-token\") pod \"cert-manager-55b4d954d5-gk7fr\" (UID: \"033e0b80-fcc8-475a-9c22-d02216700d96\") " pod="cert-manager/cert-manager-55b4d954d5-gk7fr"
Jul 07 06:35:58.573291 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:35:58.573267    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-gk7fr"]
Jul 07 06:35:58.671251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:35:58.671218    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/033e0b80-fcc8-475a-9c22-d02216700d96-bound-sa-token\") pod \"cert-manager-55b4d954d5-gk7fr\" (UID: \"033e0b80-fcc8-475a-9c22-d02216700d96\") " pod="cert-manager/cert-manager-55b4d954d5-gk7fr"
Jul 07 06:35:58.671433 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:35:58.671276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbdx7\" (UniqueName: \"kubernetes.io/projected/033e0b80-fcc8-475a-9c22-d02216700d96-kube-api-access-qbdx7\") pod \"cert-manager-55b4d954d5-gk7fr\" (UID: \"033e0b80-fcc8-475a-9c22-d02216700d96\") " pod="cert-manager/cert-manager-55b4d954d5-gk7fr"
Jul 07 06:35:58.679651 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:35:58.679601    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/033e0b80-fcc8-475a-9c22-d02216700d96-bound-sa-token\") pod \"cert-manager-55b4d954d5-gk7fr\" (UID: \"033e0b80-fcc8-475a-9c22-d02216700d96\") " pod="cert-manager/cert-manager-55b4d954d5-gk7fr"
Jul 07 06:35:58.679799 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:35:58.679739    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbdx7\" (UniqueName: \"kubernetes.io/projected/033e0b80-fcc8-475a-9c22-d02216700d96-kube-api-access-qbdx7\") pod \"cert-manager-55b4d954d5-gk7fr\" (UID: \"033e0b80-fcc8-475a-9c22-d02216700d96\") " pod="cert-manager/cert-manager-55b4d954d5-gk7fr"
Jul 07 06:35:58.891027 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:35:58.890928    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-gk7fr"
Jul 07 06:35:59.024237 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:35:59.024150    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-gk7fr"]
Jul 07 06:35:59.026882 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:35:59.026854    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod033e0b80_fcc8_475a_9c22_d02216700d96.slice/crio-17ae09d7947a9385b5b675d5f0315483bb1c873491c2ff8027a4aa54c038f07b WatchSource:0}: Error finding container 17ae09d7947a9385b5b675d5f0315483bb1c873491c2ff8027a4aa54c038f07b: Status 404 returned error can't find the container with id 17ae09d7947a9385b5b675d5f0315483bb1c873491c2ff8027a4aa54c038f07b
Jul 07 06:35:59.327903 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:35:59.327861    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-gk7fr" event={"ID":"033e0b80-fcc8-475a-9c22-d02216700d96","Type":"ContainerStarted","Data":"17ae09d7947a9385b5b675d5f0315483bb1c873491c2ff8027a4aa54c038f07b"}
Jul 07 06:36:02.343691 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:02.343624    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-gk7fr" event={"ID":"033e0b80-fcc8-475a-9c22-d02216700d96","Type":"ContainerStarted","Data":"55a4166958f60e2ec4f4daee114ea6b09248dfbb69ef8e5b719fe613fd7bf5d2"}
Jul 07 06:36:02.359911 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:02.359800    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-gk7fr" podStartSLOduration=1.278463667 podStartE2EDuration="4.359781988s" totalImagesPullingTime="3.081318321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:36:02.358927496 +0000 UTC m=+245.723124034" watchObservedRunningTime="2026-07-07 06:36:02.359781988 +0000 UTC m=+245.723978533"
Jul 07 06:36:24.807533 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:24.807452    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-856bfb9fcf-zdcjc"]
Jul 07 06:36:24.811049 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:24.811031    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-856bfb9fcf-zdcjc"
Jul 07 06:36:24.815587 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:24.815563    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 07 06:36:24.815769 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:24.815599    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 07 06:36:24.815769 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:24.815560    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 07 06:36:24.815769 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:24.815566    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 07 06:36:24.815769 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:24.815566    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 07 06:36:24.815769 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:24.815560    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-lc28m\""
Jul 07 06:36:24.824398 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:24.824373    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-856bfb9fcf-zdcjc"]
Jul 07 06:36:24.871380 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:24.871342    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64rrj\" (UniqueName: \"kubernetes.io/projected/d5ed9d72-fcb9-43a4-9cad-9a5f217b0437-kube-api-access-64rrj\") pod \"lws-controller-manager-856bfb9fcf-zdcjc\" (UID: \"d5ed9d72-fcb9-43a4-9cad-9a5f217b0437\") " pod="openshift-lws-operator/lws-controller-manager-856bfb9fcf-zdcjc"
Jul 07 06:36:24.871380 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:24.871385    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d5ed9d72-fcb9-43a4-9cad-9a5f217b0437-metrics-cert\") pod \"lws-controller-manager-856bfb9fcf-zdcjc\" (UID: \"d5ed9d72-fcb9-43a4-9cad-9a5f217b0437\") " pod="openshift-lws-operator/lws-controller-manager-856bfb9fcf-zdcjc"
Jul 07 06:36:24.871588 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:24.871447    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5ed9d72-fcb9-43a4-9cad-9a5f217b0437-cert\") pod \"lws-controller-manager-856bfb9fcf-zdcjc\" (UID: \"d5ed9d72-fcb9-43a4-9cad-9a5f217b0437\") " pod="openshift-lws-operator/lws-controller-manager-856bfb9fcf-zdcjc"
Jul 07 06:36:24.871588 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:24.871477    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d5ed9d72-fcb9-43a4-9cad-9a5f217b0437-manager-config\") pod \"lws-controller-manager-856bfb9fcf-zdcjc\" (UID: \"d5ed9d72-fcb9-43a4-9cad-9a5f217b0437\") " pod="openshift-lws-operator/lws-controller-manager-856bfb9fcf-zdcjc"
Jul 07 06:36:24.972834 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:24.972782    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64rrj\" (UniqueName: \"kubernetes.io/projected/d5ed9d72-fcb9-43a4-9cad-9a5f217b0437-kube-api-access-64rrj\") pod \"lws-controller-manager-856bfb9fcf-zdcjc\" (UID: \"d5ed9d72-fcb9-43a4-9cad-9a5f217b0437\") " pod="openshift-lws-operator/lws-controller-manager-856bfb9fcf-zdcjc"
Jul 07 06:36:24.972834 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:24.972838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d5ed9d72-fcb9-43a4-9cad-9a5f217b0437-metrics-cert\") pod \"lws-controller-manager-856bfb9fcf-zdcjc\" (UID: \"d5ed9d72-fcb9-43a4-9cad-9a5f217b0437\") " pod="openshift-lws-operator/lws-controller-manager-856bfb9fcf-zdcjc"
Jul 07 06:36:24.973062 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:24.972884    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5ed9d72-fcb9-43a4-9cad-9a5f217b0437-cert\") pod \"lws-controller-manager-856bfb9fcf-zdcjc\" (UID: \"d5ed9d72-fcb9-43a4-9cad-9a5f217b0437\") " pod="openshift-lws-operator/lws-controller-manager-856bfb9fcf-zdcjc"
Jul 07 06:36:24.973062 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:24.972908    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d5ed9d72-fcb9-43a4-9cad-9a5f217b0437-manager-config\") pod \"lws-controller-manager-856bfb9fcf-zdcjc\" (UID: \"d5ed9d72-fcb9-43a4-9cad-9a5f217b0437\") " pod="openshift-lws-operator/lws-controller-manager-856bfb9fcf-zdcjc"
Jul 07 06:36:24.973685 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:24.973666    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d5ed9d72-fcb9-43a4-9cad-9a5f217b0437-manager-config\") pod \"lws-controller-manager-856bfb9fcf-zdcjc\" (UID: \"d5ed9d72-fcb9-43a4-9cad-9a5f217b0437\") " pod="openshift-lws-operator/lws-controller-manager-856bfb9fcf-zdcjc"
Jul 07 06:36:24.975505 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:24.975489    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5ed9d72-fcb9-43a4-9cad-9a5f217b0437-cert\") pod \"lws-controller-manager-856bfb9fcf-zdcjc\" (UID: \"d5ed9d72-fcb9-43a4-9cad-9a5f217b0437\") " pod="openshift-lws-operator/lws-controller-manager-856bfb9fcf-zdcjc"
Jul 07 06:36:24.975554 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:24.975540    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d5ed9d72-fcb9-43a4-9cad-9a5f217b0437-metrics-cert\") pod \"lws-controller-manager-856bfb9fcf-zdcjc\" (UID: \"d5ed9d72-fcb9-43a4-9cad-9a5f217b0437\") " pod="openshift-lws-operator/lws-controller-manager-856bfb9fcf-zdcjc"
Jul 07 06:36:24.991331 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:24.991299    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64rrj\" (UniqueName: \"kubernetes.io/projected/d5ed9d72-fcb9-43a4-9cad-9a5f217b0437-kube-api-access-64rrj\") pod \"lws-controller-manager-856bfb9fcf-zdcjc\" (UID: \"d5ed9d72-fcb9-43a4-9cad-9a5f217b0437\") " pod="openshift-lws-operator/lws-controller-manager-856bfb9fcf-zdcjc"
Jul 07 06:36:25.120452 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:25.120362    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-856bfb9fcf-zdcjc"
Jul 07 06:36:25.258755 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:25.258729    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-856bfb9fcf-zdcjc"]
Jul 07 06:36:25.260892 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:36:25.260860    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ed9d72_fcb9_43a4_9cad_9a5f217b0437.slice/crio-b95911e7256b7dd74fc811dc359242bc661601cbed4ac339cacfa260b5acfbb6 WatchSource:0}: Error finding container b95911e7256b7dd74fc811dc359242bc661601cbed4ac339cacfa260b5acfbb6: Status 404 returned error can't find the container with id b95911e7256b7dd74fc811dc359242bc661601cbed4ac339cacfa260b5acfbb6
Jul 07 06:36:25.424349 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:25.424265    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-856bfb9fcf-zdcjc" event={"ID":"d5ed9d72-fcb9-43a4-9cad-9a5f217b0437","Type":"ContainerStarted","Data":"b95911e7256b7dd74fc811dc359242bc661601cbed4ac339cacfa260b5acfbb6"}
Jul 07 06:36:28.439928 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:28.439892    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-856bfb9fcf-zdcjc" event={"ID":"d5ed9d72-fcb9-43a4-9cad-9a5f217b0437","Type":"ContainerStarted","Data":"6a72c644122096349f5aeda64acc2048235fad3a80b8c0da22dea2905327c2d4"}
Jul 07 06:36:28.440344 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:28.440036    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-856bfb9fcf-zdcjc"
Jul 07 06:36:28.467795 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:28.467734    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-856bfb9fcf-zdcjc" podStartSLOduration=2.101190847 podStartE2EDuration="4.467719407s" totalImagesPullingTime="2.36652856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:36:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:36:28.466890479 +0000 UTC m=+271.831087024" watchObservedRunningTime="2026-07-07 06:36:28.467719407 +0000 UTC m=+271.831915951"
Jul 07 06:36:30.579139 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:30.579101    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks"]
Jul 07 06:36:30.582827 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:30.582802    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks"
Jul 07 06:36:30.593740 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:30.593713    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 07 06:36:30.593901 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:36:30.593707    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-service-cert\" is forbidden: User \"system:node:ip-10-0-133-92.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-92.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret"
Jul 07 06:36:30.599307 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:30.599278    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 07 06:36:30.599307 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:36:30.599282    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"opendatahub-operator-controller-manager-c66c8b568-dwbks\" is forbidden: User \"system:node:ip-10-0-133-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-92.ec2.internal' and this object" podUID="3de2d723-cc06-48f1-a5cb-28994950a44b" pod="opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks"
Jul 07 06:36:30.599496 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:30.599314    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 07 06:36:30.599590 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:30.599577    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mhfqd\""
Jul 07 06:36:30.608068 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:30.608035    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks"]
Jul 07 06:36:30.723520 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:30.723478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3de2d723-cc06-48f1-a5cb-28994950a44b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c66c8b568-dwbks\" (UID: \"3de2d723-cc06-48f1-a5cb-28994950a44b\") " pod="opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks"
Jul 07 06:36:30.723738 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:30.723532    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3de2d723-cc06-48f1-a5cb-28994950a44b-webhook-cert\") pod \"opendatahub-operator-controller-manager-c66c8b568-dwbks\" (UID: \"3de2d723-cc06-48f1-a5cb-28994950a44b\") " pod="opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks"
Jul 07 06:36:30.723738 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:30.723556    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3de2d723-cc06-48f1-a5cb-28994950a44b-tmp\") pod \"opendatahub-operator-controller-manager-c66c8b568-dwbks\" (UID: \"3de2d723-cc06-48f1-a5cb-28994950a44b\") " pod="opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks"
Jul 07 06:36:30.723738 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:30.723608    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3de2d723-cc06-48f1-a5cb-28994950a44b-manifests\") pod \"opendatahub-operator-controller-manager-c66c8b568-dwbks\" (UID: \"3de2d723-cc06-48f1-a5cb-28994950a44b\") " pod="opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks"
Jul 07 06:36:30.723738 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:30.723661    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbf98\" (UniqueName: \"kubernetes.io/projected/3de2d723-cc06-48f1-a5cb-28994950a44b-kube-api-access-kbf98\") pod \"opendatahub-operator-controller-manager-c66c8b568-dwbks\" (UID: \"3de2d723-cc06-48f1-a5cb-28994950a44b\") " pod="opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks"
Jul 07 06:36:30.824997 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:30.824960    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3de2d723-cc06-48f1-a5cb-28994950a44b-webhook-cert\") pod \"opendatahub-operator-controller-manager-c66c8b568-dwbks\" (UID: \"3de2d723-cc06-48f1-a5cb-28994950a44b\") " pod="opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks"
Jul 07 06:36:30.824997 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:30.824996    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3de2d723-cc06-48f1-a5cb-28994950a44b-tmp\") pod \"opendatahub-operator-controller-manager-c66c8b568-dwbks\" (UID: \"3de2d723-cc06-48f1-a5cb-28994950a44b\") " pod="opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks"
Jul 07 06:36:30.825220 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:30.825022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3de2d723-cc06-48f1-a5cb-28994950a44b-manifests\") pod \"opendatahub-operator-controller-manager-c66c8b568-dwbks\" (UID: \"3de2d723-cc06-48f1-a5cb-28994950a44b\") " pod="opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks"
Jul 07 06:36:30.825220 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:30.825045    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbf98\" (UniqueName: \"kubernetes.io/projected/3de2d723-cc06-48f1-a5cb-28994950a44b-kube-api-access-kbf98\") pod \"opendatahub-operator-controller-manager-c66c8b568-dwbks\" (UID: \"3de2d723-cc06-48f1-a5cb-28994950a44b\") " pod="opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks"
Jul 07 06:36:30.825220 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:30.825104    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3de2d723-cc06-48f1-a5cb-28994950a44b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c66c8b568-dwbks\" (UID: \"3de2d723-cc06-48f1-a5cb-28994950a44b\") " pod="opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks"
Jul 07 06:36:30.825505 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:30.825486    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3de2d723-cc06-48f1-a5cb-28994950a44b-manifests\") pod \"opendatahub-operator-controller-manager-c66c8b568-dwbks\" (UID: \"3de2d723-cc06-48f1-a5cb-28994950a44b\") " pod="opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks"
Jul 07 06:36:30.827478 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:30.827446    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3de2d723-cc06-48f1-a5cb-28994950a44b-tmp\") pod \"opendatahub-operator-controller-manager-c66c8b568-dwbks\" (UID: \"3de2d723-cc06-48f1-a5cb-28994950a44b\") " pod="opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks"
Jul 07 06:36:30.833822 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:30.833754    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbf98\" (UniqueName: \"kubernetes.io/projected/3de2d723-cc06-48f1-a5cb-28994950a44b-kube-api-access-kbf98\") pod \"opendatahub-operator-controller-manager-c66c8b568-dwbks\" (UID: \"3de2d723-cc06-48f1-a5cb-28994950a44b\") " pod="opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks"
Jul 07 06:36:31.825965 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:36:31.825910    2563 secret.go:189] Couldn't get secret opendatahub/opendatahub-operator-controller-manager-service-cert: failed to sync secret cache: timed out waiting for the condition
Jul 07 06:36:31.826384 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:36:31.826005    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3de2d723-cc06-48f1-a5cb-28994950a44b-webhook-cert podName:3de2d723-cc06-48f1-a5cb-28994950a44b nodeName:}" failed. No retries permitted until 2026-07-07 06:36:32.3259884 +0000 UTC m=+275.690184926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/3de2d723-cc06-48f1-a5cb-28994950a44b-webhook-cert") pod "opendatahub-operator-controller-manager-c66c8b568-dwbks" (UID: "3de2d723-cc06-48f1-a5cb-28994950a44b") : failed to sync secret cache: timed out waiting for the condition
Jul 07 06:36:31.826384 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:36:31.825919    2563 secret.go:189] Couldn't get secret opendatahub/opendatahub-operator-controller-manager-service-cert: failed to sync secret cache: timed out waiting for the condition
Jul 07 06:36:31.826384 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:36:31.826095    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3de2d723-cc06-48f1-a5cb-28994950a44b-apiservice-cert podName:3de2d723-cc06-48f1-a5cb-28994950a44b nodeName:}" failed. No retries permitted until 2026-07-07 06:36:32.326079169 +0000 UTC m=+275.690275691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "apiservice-cert" (UniqueName: "kubernetes.io/secret/3de2d723-cc06-48f1-a5cb-28994950a44b-apiservice-cert") pod "opendatahub-operator-controller-manager-c66c8b568-dwbks" (UID: "3de2d723-cc06-48f1-a5cb-28994950a44b") : failed to sync secret cache: timed out waiting for the condition
Jul 07 06:36:31.878523 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:31.878493    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 07 06:36:32.337792 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:32.337746    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3de2d723-cc06-48f1-a5cb-28994950a44b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c66c8b568-dwbks\" (UID: \"3de2d723-cc06-48f1-a5cb-28994950a44b\") " pod="opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks"
Jul 07 06:36:32.337976 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:32.337818    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3de2d723-cc06-48f1-a5cb-28994950a44b-webhook-cert\") pod \"opendatahub-operator-controller-manager-c66c8b568-dwbks\" (UID: \"3de2d723-cc06-48f1-a5cb-28994950a44b\") " pod="opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks"
Jul 07 06:36:32.340409 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:32.340378    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3de2d723-cc06-48f1-a5cb-28994950a44b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c66c8b568-dwbks\" (UID: \"3de2d723-cc06-48f1-a5cb-28994950a44b\") " pod="opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks"
Jul 07 06:36:32.340524 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:32.340436    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3de2d723-cc06-48f1-a5cb-28994950a44b-webhook-cert\") pod \"opendatahub-operator-controller-manager-c66c8b568-dwbks\" (UID: \"3de2d723-cc06-48f1-a5cb-28994950a44b\") " pod="opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks"
Jul 07 06:36:32.394498 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:32.394455    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks"
Jul 07 06:36:32.549787 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:32.549763    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks"]
Jul 07 06:36:32.551775 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:36:32.551744    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de2d723_cc06_48f1_a5cb_28994950a44b.slice/crio-be430093517390375b588959bb74b4be3321e8f781e8435fcaa40d05d8cfcca2 WatchSource:0}: Error finding container be430093517390375b588959bb74b4be3321e8f781e8435fcaa40d05d8cfcca2: Status 404 returned error can't find the container with id be430093517390375b588959bb74b4be3321e8f781e8435fcaa40d05d8cfcca2
Jul 07 06:36:33.470929 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:33.470893    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks" event={"ID":"3de2d723-cc06-48f1-a5cb-28994950a44b","Type":"ContainerStarted","Data":"be430093517390375b588959bb74b4be3321e8f781e8435fcaa40d05d8cfcca2"}
Jul 07 06:36:35.480589 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:35.480555    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks" event={"ID":"3de2d723-cc06-48f1-a5cb-28994950a44b","Type":"ContainerStarted","Data":"d8b19b90d5d4be44ab8beb9935746c555687bd4d3a3ef69af650711215155943"}
Jul 07 06:36:36.486538 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:36.486503    2563 generic.go:356] "Generic (PLEG): container finished" podID="3de2d723-cc06-48f1-a5cb-28994950a44b" containerID="d8b19b90d5d4be44ab8beb9935746c555687bd4d3a3ef69af650711215155943" exitCode=0
Jul 07 06:36:36.486991 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:36.486586    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks" event={"ID":"3de2d723-cc06-48f1-a5cb-28994950a44b","Type":"ContainerDied","Data":"d8b19b90d5d4be44ab8beb9935746c555687bd4d3a3ef69af650711215155943"}
Jul 07 06:36:37.492016 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:37.491985    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks" event={"ID":"3de2d723-cc06-48f1-a5cb-28994950a44b","Type":"ContainerStarted","Data":"1ee6873ab412bfb26626c0f07b00d02b118e177efda551d155f50ef6e3a3a758"}
Jul 07 06:36:37.492438 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:37.492115    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks"
Jul 07 06:36:37.514652 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:37.514582    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks" podStartSLOduration=3.290958968 podStartE2EDuration="7.514564028s" totalImagesPullingTime="3.164250142s" totalInitContainerRuntime="1.059354918s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:36:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:36:37.513304424 +0000 UTC m=+280.877500970" watchObservedRunningTime="2026-07-07 06:36:37.514564028 +0000 UTC m=+280.878760573"
Jul 07 06:36:39.446205 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:39.446177    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-856bfb9fcf-zdcjc"
Jul 07 06:36:48.498687 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:48.498656    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-c66c8b568-dwbks"
Jul 07 06:36:50.441203 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:50.441162    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-74994db979-jwcx7"]
Jul 07 06:36:50.444375 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:50.444352    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-74994db979-jwcx7"
Jul 07 06:36:50.451249 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:50.451229    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 07 06:36:50.451520 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:50.451503    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-gdh9s\""
Jul 07 06:36:50.455043 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:50.455026    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 07 06:36:50.456976 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:50.456962    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 07 06:36:50.457224 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:50.457211    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 07 06:36:50.470543 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:50.470520    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-74994db979-jwcx7"]
Jul 07 06:36:50.594023 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:50.593989    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce07d74d-6b0a-4e94-b31c-c9553c5f3a1b-tls-certs\") pod \"kube-auth-proxy-74994db979-jwcx7\" (UID: \"ce07d74d-6b0a-4e94-b31c-c9553c5f3a1b\") " pod="openshift-ingress/kube-auth-proxy-74994db979-jwcx7"
Jul 07 06:36:50.594225 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:50.594031    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxc6d\" (UniqueName: \"kubernetes.io/projected/ce07d74d-6b0a-4e94-b31c-c9553c5f3a1b-kube-api-access-dxc6d\") pod \"kube-auth-proxy-74994db979-jwcx7\" (UID: \"ce07d74d-6b0a-4e94-b31c-c9553c5f3a1b\") " pod="openshift-ingress/kube-auth-proxy-74994db979-jwcx7"
Jul 07 06:36:50.594225 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:50.594102    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce07d74d-6b0a-4e94-b31c-c9553c5f3a1b-tmp\") pod \"kube-auth-proxy-74994db979-jwcx7\" (UID: \"ce07d74d-6b0a-4e94-b31c-c9553c5f3a1b\") " pod="openshift-ingress/kube-auth-proxy-74994db979-jwcx7"
Jul 07 06:36:50.695218 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:50.695140    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce07d74d-6b0a-4e94-b31c-c9553c5f3a1b-tmp\") pod \"kube-auth-proxy-74994db979-jwcx7\" (UID: \"ce07d74d-6b0a-4e94-b31c-c9553c5f3a1b\") " pod="openshift-ingress/kube-auth-proxy-74994db979-jwcx7"
Jul 07 06:36:50.695360 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:50.695221    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce07d74d-6b0a-4e94-b31c-c9553c5f3a1b-tls-certs\") pod \"kube-auth-proxy-74994db979-jwcx7\" (UID: \"ce07d74d-6b0a-4e94-b31c-c9553c5f3a1b\") " pod="openshift-ingress/kube-auth-proxy-74994db979-jwcx7"
Jul 07 06:36:50.695360 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:50.695245    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxc6d\" (UniqueName: \"kubernetes.io/projected/ce07d74d-6b0a-4e94-b31c-c9553c5f3a1b-kube-api-access-dxc6d\") pod \"kube-auth-proxy-74994db979-jwcx7\" (UID: \"ce07d74d-6b0a-4e94-b31c-c9553c5f3a1b\") " pod="openshift-ingress/kube-auth-proxy-74994db979-jwcx7"
Jul 07 06:36:50.697684 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:50.697630    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce07d74d-6b0a-4e94-b31c-c9553c5f3a1b-tmp\") pod \"kube-auth-proxy-74994db979-jwcx7\" (UID: \"ce07d74d-6b0a-4e94-b31c-c9553c5f3a1b\") " pod="openshift-ingress/kube-auth-proxy-74994db979-jwcx7"
Jul 07 06:36:50.697807 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:50.697774    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce07d74d-6b0a-4e94-b31c-c9553c5f3a1b-tls-certs\") pod \"kube-auth-proxy-74994db979-jwcx7\" (UID: \"ce07d74d-6b0a-4e94-b31c-c9553c5f3a1b\") " pod="openshift-ingress/kube-auth-proxy-74994db979-jwcx7"
Jul 07 06:36:50.715158 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:50.715131    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxc6d\" (UniqueName: \"kubernetes.io/projected/ce07d74d-6b0a-4e94-b31c-c9553c5f3a1b-kube-api-access-dxc6d\") pod \"kube-auth-proxy-74994db979-jwcx7\" (UID: \"ce07d74d-6b0a-4e94-b31c-c9553c5f3a1b\") " pod="openshift-ingress/kube-auth-proxy-74994db979-jwcx7"
Jul 07 06:36:50.753713 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:50.753685    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-74994db979-jwcx7"
Jul 07 06:36:50.891678 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:50.891654    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-74994db979-jwcx7"]
Jul 07 06:36:50.893959 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:36:50.893931    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce07d74d_6b0a_4e94_b31c_c9553c5f3a1b.slice/crio-97e948be338f22da981a43b82645df5683f175cb4c5d5a42d8a722ca9e00a19d WatchSource:0}: Error finding container 97e948be338f22da981a43b82645df5683f175cb4c5d5a42d8a722ca9e00a19d: Status 404 returned error can't find the container with id 97e948be338f22da981a43b82645df5683f175cb4c5d5a42d8a722ca9e00a19d
Jul 07 06:36:51.547586 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:51.547551    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-74994db979-jwcx7" event={"ID":"ce07d74d-6b0a-4e94-b31c-c9553c5f3a1b","Type":"ContainerStarted","Data":"97e948be338f22da981a43b82645df5683f175cb4c5d5a42d8a722ca9e00a19d"}
Jul 07 06:36:54.565374 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:54.565342    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-74994db979-jwcx7" event={"ID":"ce07d74d-6b0a-4e94-b31c-c9553c5f3a1b","Type":"ContainerStarted","Data":"aff6dfccd3a873b031ad1b1343d1aface657a0ac8cd833300c56bdd6b0336405"}
Jul 07 06:36:54.582229 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:54.582170    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-74994db979-jwcx7" podStartSLOduration=1.605492882 podStartE2EDuration="4.582154181s" totalImagesPullingTime="2.976661299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:36:54.581464636 +0000 UTC m=+297.945661181" watchObservedRunningTime="2026-07-07 06:36:54.582154181 +0000 UTC m=+297.946350725"
Jul 07 06:36:57.053608 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:36:57.053573    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 07 06:37:02.598258 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:37:02.598223    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6hmxm"]
Jul 07 06:37:02.601354 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:37:02.601339    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6hmxm"
Jul 07 06:37:02.604086 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:37:02.604064    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-9hl4p\""
Jul 07 06:37:02.604216 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:37:02.604066    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 07 06:37:02.614078 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:37:02.614049    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6hmxm"]
Jul 07 06:37:02.697565 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:37:02.697519    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c21ff8a6-6973-47f7-a056-9d59f933a8df-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6hmxm\" (UID: \"c21ff8a6-6973-47f7-a056-9d59f933a8df\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6hmxm"
Jul 07 06:37:02.697780 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:37:02.697597    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hczkt\" (UniqueName: \"kubernetes.io/projected/c21ff8a6-6973-47f7-a056-9d59f933a8df-kube-api-access-hczkt\") pod \"llmisvc-controller-manager-777c6bd7d5-6hmxm\" (UID: \"c21ff8a6-6973-47f7-a056-9d59f933a8df\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6hmxm"
Jul 07 06:37:02.798062 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:37:02.798028    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hczkt\" (UniqueName: \"kubernetes.io/projected/c21ff8a6-6973-47f7-a056-9d59f933a8df-kube-api-access-hczkt\") pod \"llmisvc-controller-manager-777c6bd7d5-6hmxm\" (UID: \"c21ff8a6-6973-47f7-a056-9d59f933a8df\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6hmxm"
Jul 07 06:37:02.798242 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:37:02.798081    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c21ff8a6-6973-47f7-a056-9d59f933a8df-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6hmxm\" (UID: \"c21ff8a6-6973-47f7-a056-9d59f933a8df\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6hmxm"
Jul 07 06:37:02.800717 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:37:02.800698    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c21ff8a6-6973-47f7-a056-9d59f933a8df-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6hmxm\" (UID: \"c21ff8a6-6973-47f7-a056-9d59f933a8df\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6hmxm"
Jul 07 06:37:02.817060 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:37:02.817022    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hczkt\" (UniqueName: \"kubernetes.io/projected/c21ff8a6-6973-47f7-a056-9d59f933a8df-kube-api-access-hczkt\") pod \"llmisvc-controller-manager-777c6bd7d5-6hmxm\" (UID: \"c21ff8a6-6973-47f7-a056-9d59f933a8df\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6hmxm"
Jul 07 06:37:02.912136 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:37:02.912045    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6hmxm"
Jul 07 06:37:03.045712 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:37:03.045675    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6hmxm"]
Jul 07 06:37:03.051062 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:37:03.051019    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc21ff8a6_6973_47f7_a056_9d59f933a8df.slice/crio-d5b803cd4b56b067f93f7ff30e0420c042367626bfd8400ef4e55daa366aa539 WatchSource:0}: Error finding container d5b803cd4b56b067f93f7ff30e0420c042367626bfd8400ef4e55daa366aa539: Status 404 returned error can't find the container with id d5b803cd4b56b067f93f7ff30e0420c042367626bfd8400ef4e55daa366aa539
Jul 07 06:37:03.052405 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:37:03.052385    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 06:37:03.605677 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:37:03.605611    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6hmxm" event={"ID":"c21ff8a6-6973-47f7-a056-9d59f933a8df","Type":"ContainerStarted","Data":"d5b803cd4b56b067f93f7ff30e0420c042367626bfd8400ef4e55daa366aa539"}
Jul 07 06:37:06.620227 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:37:06.620193    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6hmxm" event={"ID":"c21ff8a6-6973-47f7-a056-9d59f933a8df","Type":"ContainerStarted","Data":"f1a482644bd9d9401b06a8e6950b8db7bb684a50f8ba742cf7377903d030a5c4"}
Jul 07 06:37:06.620695 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:37:06.620338    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6hmxm"
Jul 07 06:37:06.638930 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:37:06.638878    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6hmxm" podStartSLOduration=1.271380908 podStartE2EDuration="4.638860878s" totalImagesPullingTime="3.36747997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:37:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:37:06.638372618 +0000 UTC m=+310.002569195" watchObservedRunningTime="2026-07-07 06:37:06.638860878 +0000 UTC m=+310.003057488"
Jul 07 06:37:37.627204 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:37:37.627173    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6hmxm"
Jul 07 06:38:13.579049 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:13.579013    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d577798c-n7ggq"]
Jul 07 06:38:13.582677 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:13.582655    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d577798c-n7ggq"
Jul 07 06:38:13.596146 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:13.596113    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d577798c-n7ggq"]
Jul 07 06:38:13.694118 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:13.694076    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52c18bf8-c9a6-45c1-a7a5-6458f15edc6b-console-config\") pod \"console-5d577798c-n7ggq\" (UID: \"52c18bf8-c9a6-45c1-a7a5-6458f15edc6b\") " pod="openshift-console/console-5d577798c-n7ggq"
Jul 07 06:38:13.694118 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:13.694122    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52c18bf8-c9a6-45c1-a7a5-6458f15edc6b-oauth-serving-cert\") pod \"console-5d577798c-n7ggq\" (UID: \"52c18bf8-c9a6-45c1-a7a5-6458f15edc6b\") " pod="openshift-console/console-5d577798c-n7ggq"
Jul 07 06:38:13.694367 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:13.694153    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52c18bf8-c9a6-45c1-a7a5-6458f15edc6b-console-serving-cert\") pod \"console-5d577798c-n7ggq\" (UID: \"52c18bf8-c9a6-45c1-a7a5-6458f15edc6b\") " pod="openshift-console/console-5d577798c-n7ggq"
Jul 07 06:38:13.694367 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:13.694208    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52c18bf8-c9a6-45c1-a7a5-6458f15edc6b-service-ca\") pod \"console-5d577798c-n7ggq\" (UID: \"52c18bf8-c9a6-45c1-a7a5-6458f15edc6b\") " pod="openshift-console/console-5d577798c-n7ggq"
Jul 07 06:38:13.694367 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:13.694236    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52c18bf8-c9a6-45c1-a7a5-6458f15edc6b-trusted-ca-bundle\") pod \"console-5d577798c-n7ggq\" (UID: \"52c18bf8-c9a6-45c1-a7a5-6458f15edc6b\") " pod="openshift-console/console-5d577798c-n7ggq"
Jul 07 06:38:13.694367 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:13.694299    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmc5w\" (UniqueName: \"kubernetes.io/projected/52c18bf8-c9a6-45c1-a7a5-6458f15edc6b-kube-api-access-pmc5w\") pod \"console-5d577798c-n7ggq\" (UID: \"52c18bf8-c9a6-45c1-a7a5-6458f15edc6b\") " pod="openshift-console/console-5d577798c-n7ggq"
Jul 07 06:38:13.694367 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:13.694325    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52c18bf8-c9a6-45c1-a7a5-6458f15edc6b-console-oauth-config\") pod \"console-5d577798c-n7ggq\" (UID: \"52c18bf8-c9a6-45c1-a7a5-6458f15edc6b\") " pod="openshift-console/console-5d577798c-n7ggq"
Jul 07 06:38:13.795535 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:13.795497    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52c18bf8-c9a6-45c1-a7a5-6458f15edc6b-console-config\") pod \"console-5d577798c-n7ggq\" (UID: \"52c18bf8-c9a6-45c1-a7a5-6458f15edc6b\") " pod="openshift-console/console-5d577798c-n7ggq"
Jul 07 06:38:13.795535 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:13.795536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52c18bf8-c9a6-45c1-a7a5-6458f15edc6b-oauth-serving-cert\") pod \"console-5d577798c-n7ggq\" (UID: \"52c18bf8-c9a6-45c1-a7a5-6458f15edc6b\") " pod="openshift-console/console-5d577798c-n7ggq"
Jul 07 06:38:13.795777 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:13.795575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52c18bf8-c9a6-45c1-a7a5-6458f15edc6b-console-serving-cert\") pod \"console-5d577798c-n7ggq\" (UID: \"52c18bf8-c9a6-45c1-a7a5-6458f15edc6b\") " pod="openshift-console/console-5d577798c-n7ggq"
Jul 07 06:38:13.795777 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:13.795597    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52c18bf8-c9a6-45c1-a7a5-6458f15edc6b-service-ca\") pod \"console-5d577798c-n7ggq\" (UID: \"52c18bf8-c9a6-45c1-a7a5-6458f15edc6b\") " pod="openshift-console/console-5d577798c-n7ggq"
Jul 07 06:38:13.795888 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:13.795770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52c18bf8-c9a6-45c1-a7a5-6458f15edc6b-trusted-ca-bundle\") pod \"console-5d577798c-n7ggq\" (UID: \"52c18bf8-c9a6-45c1-a7a5-6458f15edc6b\") " pod="openshift-console/console-5d577798c-n7ggq"
Jul 07 06:38:13.795888 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:13.795839    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmc5w\" (UniqueName: \"kubernetes.io/projected/52c18bf8-c9a6-45c1-a7a5-6458f15edc6b-kube-api-access-pmc5w\") pod \"console-5d577798c-n7ggq\" (UID: \"52c18bf8-c9a6-45c1-a7a5-6458f15edc6b\") " pod="openshift-console/console-5d577798c-n7ggq"
Jul 07 06:38:13.796135 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:13.796021    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52c18bf8-c9a6-45c1-a7a5-6458f15edc6b-console-oauth-config\") pod \"console-5d577798c-n7ggq\" (UID: \"52c18bf8-c9a6-45c1-a7a5-6458f15edc6b\") " pod="openshift-console/console-5d577798c-n7ggq"
Jul 07 06:38:13.796308 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:13.796287    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52c18bf8-c9a6-45c1-a7a5-6458f15edc6b-oauth-serving-cert\") pod \"console-5d577798c-n7ggq\" (UID: \"52c18bf8-c9a6-45c1-a7a5-6458f15edc6b\") " pod="openshift-console/console-5d577798c-n7ggq"
Jul 07 06:38:13.796377 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:13.796335    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52c18bf8-c9a6-45c1-a7a5-6458f15edc6b-console-config\") pod \"console-5d577798c-n7ggq\" (UID: \"52c18bf8-c9a6-45c1-a7a5-6458f15edc6b\") " pod="openshift-console/console-5d577798c-n7ggq"
Jul 07 06:38:13.796377 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:13.796352    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52c18bf8-c9a6-45c1-a7a5-6458f15edc6b-service-ca\") pod \"console-5d577798c-n7ggq\" (UID: \"52c18bf8-c9a6-45c1-a7a5-6458f15edc6b\") " pod="openshift-console/console-5d577798c-n7ggq"
Jul 07 06:38:13.796613 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:13.796588    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52c18bf8-c9a6-45c1-a7a5-6458f15edc6b-trusted-ca-bundle\") pod \"console-5d577798c-n7ggq\" (UID: \"52c18bf8-c9a6-45c1-a7a5-6458f15edc6b\") " pod="openshift-console/console-5d577798c-n7ggq"
Jul 07 06:38:13.798304 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:13.798276    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52c18bf8-c9a6-45c1-a7a5-6458f15edc6b-console-serving-cert\") pod \"console-5d577798c-n7ggq\" (UID: \"52c18bf8-c9a6-45c1-a7a5-6458f15edc6b\") " pod="openshift-console/console-5d577798c-n7ggq"
Jul 07 06:38:13.798702 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:13.798680    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52c18bf8-c9a6-45c1-a7a5-6458f15edc6b-console-oauth-config\") pod \"console-5d577798c-n7ggq\" (UID: \"52c18bf8-c9a6-45c1-a7a5-6458f15edc6b\") " pod="openshift-console/console-5d577798c-n7ggq"
Jul 07 06:38:13.804359 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:13.804333    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmc5w\" (UniqueName: \"kubernetes.io/projected/52c18bf8-c9a6-45c1-a7a5-6458f15edc6b-kube-api-access-pmc5w\") pod \"console-5d577798c-n7ggq\" (UID: \"52c18bf8-c9a6-45c1-a7a5-6458f15edc6b\") " pod="openshift-console/console-5d577798c-n7ggq"
Jul 07 06:38:13.892589 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:13.892475    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d577798c-n7ggq"
Jul 07 06:38:14.033750 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:14.033723    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d577798c-n7ggq"]
Jul 07 06:38:14.036075 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:38:14.036040    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c18bf8_c9a6_45c1_a7a5_6458f15edc6b.slice/crio-f9c9a7018741659e9e03c61bf4de3b4f470681178d2fc98ea9b74bb21957869d WatchSource:0}: Error finding container f9c9a7018741659e9e03c61bf4de3b4f470681178d2fc98ea9b74bb21957869d: Status 404 returned error can't find the container with id f9c9a7018741659e9e03c61bf4de3b4f470681178d2fc98ea9b74bb21957869d
Jul 07 06:38:14.903948 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:14.903905    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d577798c-n7ggq" event={"ID":"52c18bf8-c9a6-45c1-a7a5-6458f15edc6b","Type":"ContainerStarted","Data":"69c12cc415e91ab7219d0d8fb8b31f242fca624b722ca7e815427c987a0a1601"}
Jul 07 06:38:14.904350 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:14.903970    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d577798c-n7ggq" event={"ID":"52c18bf8-c9a6-45c1-a7a5-6458f15edc6b","Type":"ContainerStarted","Data":"f9c9a7018741659e9e03c61bf4de3b4f470681178d2fc98ea9b74bb21957869d"}
Jul 07 06:38:14.922479 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:14.922418    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d577798c-n7ggq" podStartSLOduration=1.9224007570000001 podStartE2EDuration="1.922400757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:38:14.921326523 +0000 UTC m=+378.285523084" watchObservedRunningTime="2026-07-07 06:38:14.922400757 +0000 UTC m=+378.286597371"
Jul 07 06:38:23.892617 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:23.892572    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d577798c-n7ggq"
Jul 07 06:38:23.893107 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:23.892682    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d577798c-n7ggq"
Jul 07 06:38:23.897561 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:23.897527    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d577798c-n7ggq"
Jul 07 06:38:23.946169 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:23.946134    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d577798c-n7ggq"
Jul 07 06:38:24.011812 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:24.011773    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-799b758f7c-k7jp8"]
Jul 07 06:38:41.766077 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:41.766013    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-zzcpc"]
Jul 07 06:38:41.769586 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:41.769552    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zzcpc"
Jul 07 06:38:41.772048 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:41.772007    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jxfbm\""
Jul 07 06:38:41.772272 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:41.772050    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 07 06:38:41.772272 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:41.772230    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 07 06:38:41.772576 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:41.772559    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 07 06:38:41.772775 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:41.772753    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 07 06:38:41.780067 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:41.780031    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-zzcpc"]
Jul 07 06:38:41.955960 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:41.955925    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fcdc7918-ea72-4e20-b93d-e41e6c86480e-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-zzcpc\" (UID: \"fcdc7918-ea72-4e20-b93d-e41e6c86480e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zzcpc"
Jul 07 06:38:41.955960 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:41.955960    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prszp\" (UniqueName: \"kubernetes.io/projected/fcdc7918-ea72-4e20-b93d-e41e6c86480e-kube-api-access-prszp\") pod \"kuadrant-console-plugin-66b5774985-zzcpc\" (UID: \"fcdc7918-ea72-4e20-b93d-e41e6c86480e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zzcpc"
Jul 07 06:38:41.956179 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:41.955985    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fcdc7918-ea72-4e20-b93d-e41e6c86480e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zzcpc\" (UID: \"fcdc7918-ea72-4e20-b93d-e41e6c86480e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zzcpc"
Jul 07 06:38:42.057119 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:42.057083    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fcdc7918-ea72-4e20-b93d-e41e6c86480e-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-zzcpc\" (UID: \"fcdc7918-ea72-4e20-b93d-e41e6c86480e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zzcpc"
Jul 07 06:38:42.057284 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:42.057127    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prszp\" (UniqueName: \"kubernetes.io/projected/fcdc7918-ea72-4e20-b93d-e41e6c86480e-kube-api-access-prszp\") pod \"kuadrant-console-plugin-66b5774985-zzcpc\" (UID: \"fcdc7918-ea72-4e20-b93d-e41e6c86480e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zzcpc"
Jul 07 06:38:42.057284 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:42.057148    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fcdc7918-ea72-4e20-b93d-e41e6c86480e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zzcpc\" (UID: \"fcdc7918-ea72-4e20-b93d-e41e6c86480e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zzcpc"
Jul 07 06:38:42.057748 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:42.057712    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fcdc7918-ea72-4e20-b93d-e41e6c86480e-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-zzcpc\" (UID: \"fcdc7918-ea72-4e20-b93d-e41e6c86480e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zzcpc"
Jul 07 06:38:42.059732 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:42.059714    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fcdc7918-ea72-4e20-b93d-e41e6c86480e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zzcpc\" (UID: \"fcdc7918-ea72-4e20-b93d-e41e6c86480e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zzcpc"
Jul 07 06:38:42.067751 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:42.067726    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prszp\" (UniqueName: \"kubernetes.io/projected/fcdc7918-ea72-4e20-b93d-e41e6c86480e-kube-api-access-prszp\") pod \"kuadrant-console-plugin-66b5774985-zzcpc\" (UID: \"fcdc7918-ea72-4e20-b93d-e41e6c86480e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zzcpc"
Jul 07 06:38:42.082541 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:42.082514    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zzcpc"
Jul 07 06:38:42.220605 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:42.220582    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-zzcpc"]
Jul 07 06:38:42.222681 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:38:42.222652    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcdc7918_ea72_4e20_b93d_e41e6c86480e.slice/crio-1e63c13995879e133fb692a337ae19811378a6a0032bd5b2359b9da7497763f5 WatchSource:0}: Error finding container 1e63c13995879e133fb692a337ae19811378a6a0032bd5b2359b9da7497763f5: Status 404 returned error can't find the container with id 1e63c13995879e133fb692a337ae19811378a6a0032bd5b2359b9da7497763f5
Jul 07 06:38:43.034494 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:43.034452    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zzcpc" event={"ID":"fcdc7918-ea72-4e20-b93d-e41e6c86480e","Type":"ContainerStarted","Data":"1e63c13995879e133fb692a337ae19811378a6a0032bd5b2359b9da7497763f5"}
Jul 07 06:38:49.033285 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:49.033228    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-799b758f7c-k7jp8" podUID="009a488d-d184-4adc-a788-7cfc4991d906" containerName="console" containerID="cri-o://c25ccaabb7f2eb53fbe052834d82ced611d9d8f047c600655e4ccc17b035d0f3" gracePeriod=15
Jul 07 06:38:49.289974 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:49.289907    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:38:49.324903 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:49.324870    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/009a488d-d184-4adc-a788-7cfc4991d906-oauth-serving-cert\") pod \"009a488d-d184-4adc-a788-7cfc4991d906\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") "
Jul 07 06:38:49.325101 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:49.324942    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/009a488d-d184-4adc-a788-7cfc4991d906-service-ca\") pod \"009a488d-d184-4adc-a788-7cfc4991d906\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") "
Jul 07 06:38:49.325101 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:49.324973    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/009a488d-d184-4adc-a788-7cfc4991d906-console-serving-cert\") pod \"009a488d-d184-4adc-a788-7cfc4991d906\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") "
Jul 07 06:38:49.325101 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:49.324996    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/009a488d-d184-4adc-a788-7cfc4991d906-trusted-ca-bundle\") pod \"009a488d-d184-4adc-a788-7cfc4991d906\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") "
Jul 07 06:38:49.325101 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:49.325027    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb4bz\" (UniqueName: \"kubernetes.io/projected/009a488d-d184-4adc-a788-7cfc4991d906-kube-api-access-gb4bz\") pod \"009a488d-d184-4adc-a788-7cfc4991d906\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") "
Jul 07 06:38:49.325101 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:49.325088    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/009a488d-d184-4adc-a788-7cfc4991d906-console-config\") pod \"009a488d-d184-4adc-a788-7cfc4991d906\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") "
Jul 07 06:38:49.325428 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:49.325138    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/009a488d-d184-4adc-a788-7cfc4991d906-console-oauth-config\") pod \"009a488d-d184-4adc-a788-7cfc4991d906\" (UID: \"009a488d-d184-4adc-a788-7cfc4991d906\") "
Jul 07 06:38:49.326033 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:49.325932    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/009a488d-d184-4adc-a788-7cfc4991d906-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "009a488d-d184-4adc-a788-7cfc4991d906" (UID: "009a488d-d184-4adc-a788-7cfc4991d906"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 06:38:49.326164 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:49.326028    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/009a488d-d184-4adc-a788-7cfc4991d906-service-ca" (OuterVolumeSpecName: "service-ca") pod "009a488d-d184-4adc-a788-7cfc4991d906" (UID: "009a488d-d184-4adc-a788-7cfc4991d906"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 06:38:49.326164 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:49.326122    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/009a488d-d184-4adc-a788-7cfc4991d906-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "009a488d-d184-4adc-a788-7cfc4991d906" (UID: "009a488d-d184-4adc-a788-7cfc4991d906"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 06:38:49.326164 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:49.326153    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/009a488d-d184-4adc-a788-7cfc4991d906-console-config" (OuterVolumeSpecName: "console-config") pod "009a488d-d184-4adc-a788-7cfc4991d906" (UID: "009a488d-d184-4adc-a788-7cfc4991d906"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 06:38:49.328386 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:49.328347    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/009a488d-d184-4adc-a788-7cfc4991d906-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "009a488d-d184-4adc-a788-7cfc4991d906" (UID: "009a488d-d184-4adc-a788-7cfc4991d906"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 06:38:49.328729 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:49.328596    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/009a488d-d184-4adc-a788-7cfc4991d906-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "009a488d-d184-4adc-a788-7cfc4991d906" (UID: "009a488d-d184-4adc-a788-7cfc4991d906"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 06:38:49.328810 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:49.328765    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/009a488d-d184-4adc-a788-7cfc4991d906-kube-api-access-gb4bz" (OuterVolumeSpecName: "kube-api-access-gb4bz") pod "009a488d-d184-4adc-a788-7cfc4991d906" (UID: "009a488d-d184-4adc-a788-7cfc4991d906"). InnerVolumeSpecName "kube-api-access-gb4bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 06:38:49.426060 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:49.426022    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/009a488d-d184-4adc-a788-7cfc4991d906-console-config\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:38:49.426060 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:49.426055    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/009a488d-d184-4adc-a788-7cfc4991d906-console-oauth-config\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:38:49.426060 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:49.426067    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/009a488d-d184-4adc-a788-7cfc4991d906-oauth-serving-cert\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:38:49.426345 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:49.426075    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/009a488d-d184-4adc-a788-7cfc4991d906-service-ca\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:38:49.426345 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:49.426084    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/009a488d-d184-4adc-a788-7cfc4991d906-console-serving-cert\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:38:49.426345 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:49.426107    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/009a488d-d184-4adc-a788-7cfc4991d906-trusted-ca-bundle\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:38:49.426345 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:49.426123    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb4bz\" (UniqueName: \"kubernetes.io/projected/009a488d-d184-4adc-a788-7cfc4991d906-kube-api-access-gb4bz\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:38:50.070678 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:50.070616    2563 generic.go:356] "Generic (PLEG): container finished" podID="009a488d-d184-4adc-a788-7cfc4991d906" containerID="c25ccaabb7f2eb53fbe052834d82ced611d9d8f047c600655e4ccc17b035d0f3" exitCode=0
Jul 07 06:38:50.071078 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:50.070695    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-799b758f7c-k7jp8"
Jul 07 06:38:50.071078 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:50.070705    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-799b758f7c-k7jp8" event={"ID":"009a488d-d184-4adc-a788-7cfc4991d906","Type":"ContainerDied","Data":"c25ccaabb7f2eb53fbe052834d82ced611d9d8f047c600655e4ccc17b035d0f3"}
Jul 07 06:38:50.071078 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:50.070763    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-799b758f7c-k7jp8" event={"ID":"009a488d-d184-4adc-a788-7cfc4991d906","Type":"ContainerDied","Data":"016b48508c97425e858e500059fdd1e9026e9402f744521fa8a8c8558005f374"}
Jul 07 06:38:50.071078 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:50.070783    2563 scope.go:117] "RemoveContainer" containerID="c25ccaabb7f2eb53fbe052834d82ced611d9d8f047c600655e4ccc17b035d0f3"
Jul 07 06:38:50.079947 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:50.079920    2563 scope.go:117] "RemoveContainer" containerID="c25ccaabb7f2eb53fbe052834d82ced611d9d8f047c600655e4ccc17b035d0f3"
Jul 07 06:38:50.082260 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:38:50.082235    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c25ccaabb7f2eb53fbe052834d82ced611d9d8f047c600655e4ccc17b035d0f3\": container with ID starting with c25ccaabb7f2eb53fbe052834d82ced611d9d8f047c600655e4ccc17b035d0f3 not found: ID does not exist" containerID="c25ccaabb7f2eb53fbe052834d82ced611d9d8f047c600655e4ccc17b035d0f3"
Jul 07 06:38:50.082404 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:50.082265    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c25ccaabb7f2eb53fbe052834d82ced611d9d8f047c600655e4ccc17b035d0f3"} err="failed to get container status \"c25ccaabb7f2eb53fbe052834d82ced611d9d8f047c600655e4ccc17b035d0f3\": rpc error: code = NotFound desc = could not find container \"c25ccaabb7f2eb53fbe052834d82ced611d9d8f047c600655e4ccc17b035d0f3\": container with ID starting with c25ccaabb7f2eb53fbe052834d82ced611d9d8f047c600655e4ccc17b035d0f3 not found: ID does not exist"
Jul 07 06:38:50.090007 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:50.089979    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-799b758f7c-k7jp8"]
Jul 07 06:38:50.094525 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:50.094501    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-799b758f7c-k7jp8"]
Jul 07 06:38:51.175266 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:51.175218    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="009a488d-d184-4adc-a788-7cfc4991d906" path="/var/lib/kubelet/pods/009a488d-d184-4adc-a788-7cfc4991d906/volumes"
Jul 07 06:38:51.688587 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:51.688551    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mbs6x"]
Jul 07 06:38:51.689012 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:51.688996    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="009a488d-d184-4adc-a788-7cfc4991d906" containerName="console"
Jul 07 06:38:51.689012 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:51.689013    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="009a488d-d184-4adc-a788-7cfc4991d906" containerName="console"
Jul 07 06:38:51.689149 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:51.689065    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="009a488d-d184-4adc-a788-7cfc4991d906" containerName="console"
Jul 07 06:38:51.693883 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:51.693854    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mbs6x"
Jul 07 06:38:51.696313 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:51.696289    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8kr6t\""
Jul 07 06:38:51.717512 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:51.717475    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mbs6x"]
Jul 07 06:38:51.746316 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:51.746264    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmvb4\" (UniqueName: \"kubernetes.io/projected/0f80b0c7-d35b-4116-9358-153b099771f7-kube-api-access-fmvb4\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-mbs6x\" (UID: \"0f80b0c7-d35b-4116-9358-153b099771f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mbs6x"
Jul 07 06:38:51.746499 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:51.746321    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f80b0c7-d35b-4116-9358-153b099771f7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-mbs6x\" (UID: \"0f80b0c7-d35b-4116-9358-153b099771f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mbs6x"
Jul 07 06:38:51.800665 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:51.800617    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mbs6x"]
Jul 07 06:38:51.800943 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:38:51.800921    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-fmvb4], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mbs6x" podUID="0f80b0c7-d35b-4116-9358-153b099771f7"
Jul 07 06:38:51.847863 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:51.847821    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmvb4\" (UniqueName: \"kubernetes.io/projected/0f80b0c7-d35b-4116-9358-153b099771f7-kube-api-access-fmvb4\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-mbs6x\" (UID: \"0f80b0c7-d35b-4116-9358-153b099771f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mbs6x"
Jul 07 06:38:51.848043 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:51.847887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f80b0c7-d35b-4116-9358-153b099771f7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-mbs6x\" (UID: \"0f80b0c7-d35b-4116-9358-153b099771f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mbs6x"
Jul 07 06:38:51.848331 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:51.848309    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f80b0c7-d35b-4116-9358-153b099771f7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-mbs6x\" (UID: \"0f80b0c7-d35b-4116-9358-153b099771f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mbs6x"
Jul 07 06:38:51.864986 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:51.864960    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmvb4\" (UniqueName: \"kubernetes.io/projected/0f80b0c7-d35b-4116-9358-153b099771f7-kube-api-access-fmvb4\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-mbs6x\" (UID: \"0f80b0c7-d35b-4116-9358-153b099771f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mbs6x"
Jul 07 06:38:52.081855 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:52.081821    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mbs6x"
Jul 07 06:38:52.089440 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:52.089409    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mbs6x"
Jul 07 06:38:52.149967 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:52.149924    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmvb4\" (UniqueName: \"kubernetes.io/projected/0f80b0c7-d35b-4116-9358-153b099771f7-kube-api-access-fmvb4\") pod \"0f80b0c7-d35b-4116-9358-153b099771f7\" (UID: \"0f80b0c7-d35b-4116-9358-153b099771f7\") "
Jul 07 06:38:52.150186 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:52.150021    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f80b0c7-d35b-4116-9358-153b099771f7-extensions-socket-volume\") pod \"0f80b0c7-d35b-4116-9358-153b099771f7\" (UID: \"0f80b0c7-d35b-4116-9358-153b099771f7\") "
Jul 07 06:38:52.150310 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:52.150284    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f80b0c7-d35b-4116-9358-153b099771f7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0f80b0c7-d35b-4116-9358-153b099771f7" (UID: "0f80b0c7-d35b-4116-9358-153b099771f7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 06:38:52.152630 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:52.152590    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f80b0c7-d35b-4116-9358-153b099771f7-kube-api-access-fmvb4" (OuterVolumeSpecName: "kube-api-access-fmvb4") pod "0f80b0c7-d35b-4116-9358-153b099771f7" (UID: "0f80b0c7-d35b-4116-9358-153b099771f7"). InnerVolumeSpecName "kube-api-access-fmvb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 06:38:52.251412 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:52.251361    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f80b0c7-d35b-4116-9358-153b099771f7-extensions-socket-volume\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:38:52.251412 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:52.251403    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmvb4\" (UniqueName: \"kubernetes.io/projected/0f80b0c7-d35b-4116-9358-153b099771f7-kube-api-access-fmvb4\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:38:52.290680 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:52.290619    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mbs6x"]
Jul 07 06:38:52.302025 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:52.301987    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mbs6x"]
Jul 07 06:38:52.341897 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:52.341823    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sxm7f"]
Jul 07 06:38:52.345123 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:52.345104    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sxm7f"
Jul 07 06:38:52.369190 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:52.369154    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sxm7f"]
Jul 07 06:38:52.452449 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:52.452397    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tnw9\" (UniqueName: \"kubernetes.io/projected/5ed1b7b9-af4c-4572-baec-0b96b86e4959-kube-api-access-6tnw9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sxm7f\" (UID: \"5ed1b7b9-af4c-4572-baec-0b96b86e4959\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sxm7f"
Jul 07 06:38:52.452649 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:52.452614    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ed1b7b9-af4c-4572-baec-0b96b86e4959-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sxm7f\" (UID: \"5ed1b7b9-af4c-4572-baec-0b96b86e4959\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sxm7f"
Jul 07 06:38:52.553760 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:52.553715    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ed1b7b9-af4c-4572-baec-0b96b86e4959-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sxm7f\" (UID: \"5ed1b7b9-af4c-4572-baec-0b96b86e4959\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sxm7f"
Jul 07 06:38:52.554012 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:52.553831    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tnw9\" (UniqueName: \"kubernetes.io/projected/5ed1b7b9-af4c-4572-baec-0b96b86e4959-kube-api-access-6tnw9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sxm7f\" (UID: \"5ed1b7b9-af4c-4572-baec-0b96b86e4959\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sxm7f"
Jul 07 06:38:52.554166 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:52.554137    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ed1b7b9-af4c-4572-baec-0b96b86e4959-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sxm7f\" (UID: \"5ed1b7b9-af4c-4572-baec-0b96b86e4959\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sxm7f"
Jul 07 06:38:52.567216 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:52.567181    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tnw9\" (UniqueName: \"kubernetes.io/projected/5ed1b7b9-af4c-4572-baec-0b96b86e4959-kube-api-access-6tnw9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sxm7f\" (UID: \"5ed1b7b9-af4c-4572-baec-0b96b86e4959\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sxm7f"
Jul 07 06:38:52.658270 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:52.658167    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sxm7f"
Jul 07 06:38:52.819628 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:52.819595    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sxm7f"]
Jul 07 06:38:52.823622 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:38:52.823597    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ed1b7b9_af4c_4572_baec_0b96b86e4959.slice/crio-d522118e0c52869fb4122147715c6f3fe0896a3b7b9a5813ec318a53f9d279bc WatchSource:0}: Error finding container d522118e0c52869fb4122147715c6f3fe0896a3b7b9a5813ec318a53f9d279bc: Status 404 returned error can't find the container with id d522118e0c52869fb4122147715c6f3fe0896a3b7b9a5813ec318a53f9d279bc
Jul 07 06:38:53.089090 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:53.089054    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sxm7f" event={"ID":"5ed1b7b9-af4c-4572-baec-0b96b86e4959","Type":"ContainerStarted","Data":"d522118e0c52869fb4122147715c6f3fe0896a3b7b9a5813ec318a53f9d279bc"}
Jul 07 06:38:53.089090 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:53.089076    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mbs6x"
Jul 07 06:38:53.091861 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:38:53.091825    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-mbs6x\" is forbidden: User \"system:node:ip-10-0-133-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-92.ec2.internal' and this object" podUID="0f80b0c7-d35b-4116-9358-153b099771f7" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mbs6x"
Jul 07 06:38:53.098549 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:38:53.098520    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-mbs6x\" is forbidden: User \"system:node:ip-10-0-133-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-92.ec2.internal' and this object" podUID="0f80b0c7-d35b-4116-9358-153b099771f7" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-mbs6x"
Jul 07 06:38:53.173188 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:38:53.173146    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f80b0c7-d35b-4116-9358-153b099771f7" path="/var/lib/kubelet/pods/0f80b0c7-d35b-4116-9358-153b099771f7/volumes"
Jul 07 06:39:04.691934 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:04.691898    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sxm7f"]
Jul 07 06:39:08.163185 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:08.163143    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zzcpc" event={"ID":"fcdc7918-ea72-4e20-b93d-e41e6c86480e","Type":"ContainerStarted","Data":"4615f5e4ab27a253849d8a6354469f179e84e9390b2f2fd320e225eed43aff17"}
Jul 07 06:39:08.180374 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:08.180298    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zzcpc" podStartSLOduration=1.722423177 podStartE2EDuration="27.180277682s" totalImagesPullingTime="25.457854505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:38:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:39:08.178907168 +0000 UTC m=+431.543103733" watchObservedRunningTime="2026-07-07 06:39:08.180277682 +0000 UTC m=+431.544474228"
Jul 07 06:39:12.183106 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:12.183074    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sxm7f" event={"ID":"5ed1b7b9-af4c-4572-baec-0b96b86e4959","Type":"ContainerStarted","Data":"03deea6af298130312150a2be74df1e702936e1161811775672151a4ae466e31"}
Jul 07 06:39:12.183621 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:12.183115    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sxm7f" podUID="5ed1b7b9-af4c-4572-baec-0b96b86e4959" containerName="manager" containerID="cri-o://03deea6af298130312150a2be74df1e702936e1161811775672151a4ae466e31" gracePeriod=10
Jul 07 06:39:12.183621 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:12.183191    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sxm7f"
Jul 07 06:39:12.202740 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:12.202680    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sxm7f" podStartSLOduration=1.049836039 podStartE2EDuration="20.202657419s" totalImagesPullingTime="19.15282138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:39:12.200611584 +0000 UTC m=+435.564808151" watchObservedRunningTime="2026-07-07 06:39:12.202657419 +0000 UTC m=+435.566853964"
Jul 07 06:39:12.430145 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:12.430113    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sxm7f"
Jul 07 06:39:12.549136 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:12.549046    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tnw9\" (UniqueName: \"kubernetes.io/projected/5ed1b7b9-af4c-4572-baec-0b96b86e4959-kube-api-access-6tnw9\") pod \"5ed1b7b9-af4c-4572-baec-0b96b86e4959\" (UID: \"5ed1b7b9-af4c-4572-baec-0b96b86e4959\") "
Jul 07 06:39:12.549284 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:12.549156    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ed1b7b9-af4c-4572-baec-0b96b86e4959-extensions-socket-volume\") pod \"5ed1b7b9-af4c-4572-baec-0b96b86e4959\" (UID: \"5ed1b7b9-af4c-4572-baec-0b96b86e4959\") "
Jul 07 06:39:12.549442 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:12.549417    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed1b7b9-af4c-4572-baec-0b96b86e4959-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5ed1b7b9-af4c-4572-baec-0b96b86e4959" (UID: "5ed1b7b9-af4c-4572-baec-0b96b86e4959"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 06:39:12.551508 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:12.551484    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed1b7b9-af4c-4572-baec-0b96b86e4959-kube-api-access-6tnw9" (OuterVolumeSpecName: "kube-api-access-6tnw9") pod "5ed1b7b9-af4c-4572-baec-0b96b86e4959" (UID: "5ed1b7b9-af4c-4572-baec-0b96b86e4959"). InnerVolumeSpecName "kube-api-access-6tnw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 06:39:12.649849 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:12.649812    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ed1b7b9-af4c-4572-baec-0b96b86e4959-extensions-socket-volume\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:39:12.649849 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:12.649842    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tnw9\" (UniqueName: \"kubernetes.io/projected/5ed1b7b9-af4c-4572-baec-0b96b86e4959-kube-api-access-6tnw9\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:39:13.188763 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:13.188727    2563 generic.go:356] "Generic (PLEG): container finished" podID="5ed1b7b9-af4c-4572-baec-0b96b86e4959" containerID="03deea6af298130312150a2be74df1e702936e1161811775672151a4ae466e31" exitCode=2
Jul 07 06:39:13.189137 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:13.188776    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sxm7f"
Jul 07 06:39:13.189137 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:13.188819    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sxm7f" event={"ID":"5ed1b7b9-af4c-4572-baec-0b96b86e4959","Type":"ContainerDied","Data":"03deea6af298130312150a2be74df1e702936e1161811775672151a4ae466e31"}
Jul 07 06:39:13.189137 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:13.188869    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sxm7f" event={"ID":"5ed1b7b9-af4c-4572-baec-0b96b86e4959","Type":"ContainerDied","Data":"d522118e0c52869fb4122147715c6f3fe0896a3b7b9a5813ec318a53f9d279bc"}
Jul 07 06:39:13.189137 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:13.188890    2563 scope.go:117] "RemoveContainer" containerID="03deea6af298130312150a2be74df1e702936e1161811775672151a4ae466e31"
Jul 07 06:39:13.197703 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:13.197685    2563 scope.go:117] "RemoveContainer" containerID="03deea6af298130312150a2be74df1e702936e1161811775672151a4ae466e31"
Jul 07 06:39:13.197984 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:39:13.197964    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03deea6af298130312150a2be74df1e702936e1161811775672151a4ae466e31\": container with ID starting with 03deea6af298130312150a2be74df1e702936e1161811775672151a4ae466e31 not found: ID does not exist" containerID="03deea6af298130312150a2be74df1e702936e1161811775672151a4ae466e31"
Jul 07 06:39:13.198026 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:13.197994    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03deea6af298130312150a2be74df1e702936e1161811775672151a4ae466e31"} err="failed to get container status \"03deea6af298130312150a2be74df1e702936e1161811775672151a4ae466e31\": rpc error: code = NotFound desc = could not find container \"03deea6af298130312150a2be74df1e702936e1161811775672151a4ae466e31\": container with ID starting with 03deea6af298130312150a2be74df1e702936e1161811775672151a4ae466e31 not found: ID does not exist"
Jul 07 06:39:13.206961 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:13.206935    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sxm7f"]
Jul 07 06:39:13.212520 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:13.212492    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sxm7f"]
Jul 07 06:39:15.171683 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:15.171650    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ed1b7b9-af4c-4572-baec-0b96b86e4959" path="/var/lib/kubelet/pods/5ed1b7b9-af4c-4572-baec-0b96b86e4959/volumes"
Jul 07 06:39:35.644305 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:35.644223    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-g8q6w"]
Jul 07 06:39:35.644806 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:35.644523    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ed1b7b9-af4c-4572-baec-0b96b86e4959" containerName="manager"
Jul 07 06:39:35.644806 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:35.644534    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ed1b7b9-af4c-4572-baec-0b96b86e4959" containerName="manager"
Jul 07 06:39:35.644806 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:35.644592    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ed1b7b9-af4c-4572-baec-0b96b86e4959" containerName="manager"
Jul 07 06:39:35.718048 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:35.718016    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-g8q6w"]
Jul 07 06:39:35.718236 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:35.718147    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-g8q6w"
Jul 07 06:39:35.720437 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:35.720406    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 07 06:39:35.749102 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:35.749069    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-g8q6w"]
Jul 07 06:39:35.752760 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:35.752734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d-config-file\") pod \"limitador-limitador-64c94fd64f-g8q6w\" (UID: \"c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-g8q6w"
Jul 07 06:39:35.752878 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:35.752794    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w52vc\" (UniqueName: \"kubernetes.io/projected/c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d-kube-api-access-w52vc\") pod \"limitador-limitador-64c94fd64f-g8q6w\" (UID: \"c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-g8q6w"
Jul 07 06:39:35.854342 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:35.854303    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w52vc\" (UniqueName: \"kubernetes.io/projected/c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d-kube-api-access-w52vc\") pod \"limitador-limitador-64c94fd64f-g8q6w\" (UID: \"c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-g8q6w"
Jul 07 06:39:35.854517 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:35.854421    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d-config-file\") pod \"limitador-limitador-64c94fd64f-g8q6w\" (UID: \"c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-g8q6w"
Jul 07 06:39:35.855441 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:35.855145    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d-config-file\") pod \"limitador-limitador-64c94fd64f-g8q6w\" (UID: \"c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-g8q6w"
Jul 07 06:39:35.863036 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:35.863011    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w52vc\" (UniqueName: \"kubernetes.io/projected/c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d-kube-api-access-w52vc\") pod \"limitador-limitador-64c94fd64f-g8q6w\" (UID: \"c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-g8q6w"
Jul 07 06:39:36.029785 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:36.029704    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-g8q6w"
Jul 07 06:39:36.183080 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:36.183051    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-g8q6w"]
Jul 07 06:39:36.185744 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:39:36.185718    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d33a7f_f3f3_4c2b_b88c_55448ebecf2d.slice/crio-99e0f67fbe2d33a98b4207da53c8554d9f6ba57097f2fbea8fcb95e18d589000 WatchSource:0}: Error finding container 99e0f67fbe2d33a98b4207da53c8554d9f6ba57097f2fbea8fcb95e18d589000: Status 404 returned error can't find the container with id 99e0f67fbe2d33a98b4207da53c8554d9f6ba57097f2fbea8fcb95e18d589000
Jul 07 06:39:36.286749 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:36.286664    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-g8q6w" event={"ID":"c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d","Type":"ContainerStarted","Data":"99e0f67fbe2d33a98b4207da53c8554d9f6ba57097f2fbea8fcb95e18d589000"}
Jul 07 06:39:36.429308 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:36.429277    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-rm2nb"]
Jul 07 06:39:36.546316 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:36.546197    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-rm2nb"]
Jul 07 06:39:36.546316 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:36.546286    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rm2nb"
Jul 07 06:39:36.548456 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:36.548441    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-k296v\""
Jul 07 06:39:36.561395 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:36.561370    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvdxd\" (UniqueName: \"kubernetes.io/projected/9e15e92a-9c42-4f51-91eb-0bf48eda4502-kube-api-access-tvdxd\") pod \"authorino-58cd955675-rm2nb\" (UID: \"9e15e92a-9c42-4f51-91eb-0bf48eda4502\") " pod="kuadrant-system/authorino-58cd955675-rm2nb"
Jul 07 06:39:36.662904 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:36.662864    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvdxd\" (UniqueName: \"kubernetes.io/projected/9e15e92a-9c42-4f51-91eb-0bf48eda4502-kube-api-access-tvdxd\") pod \"authorino-58cd955675-rm2nb\" (UID: \"9e15e92a-9c42-4f51-91eb-0bf48eda4502\") " pod="kuadrant-system/authorino-58cd955675-rm2nb"
Jul 07 06:39:36.671472 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:36.671442    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvdxd\" (UniqueName: \"kubernetes.io/projected/9e15e92a-9c42-4f51-91eb-0bf48eda4502-kube-api-access-tvdxd\") pod \"authorino-58cd955675-rm2nb\" (UID: \"9e15e92a-9c42-4f51-91eb-0bf48eda4502\") " pod="kuadrant-system/authorino-58cd955675-rm2nb"
Jul 07 06:39:36.855650 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:36.855601    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rm2nb"
Jul 07 06:39:36.993496 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:36.993449    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-rm2nb"]
Jul 07 06:39:36.996596 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:39:36.996563    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e15e92a_9c42_4f51_91eb_0bf48eda4502.slice/crio-d81822b57333b92974928668dc231bd1f2cc533120d8c83f9da7779373ec30a9 WatchSource:0}: Error finding container d81822b57333b92974928668dc231bd1f2cc533120d8c83f9da7779373ec30a9: Status 404 returned error can't find the container with id d81822b57333b92974928668dc231bd1f2cc533120d8c83f9da7779373ec30a9
Jul 07 06:39:37.293480 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:37.293396    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rm2nb" event={"ID":"9e15e92a-9c42-4f51-91eb-0bf48eda4502","Type":"ContainerStarted","Data":"d81822b57333b92974928668dc231bd1f2cc533120d8c83f9da7779373ec30a9"}
Jul 07 06:39:41.315435 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:41.315392    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-g8q6w" event={"ID":"c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d","Type":"ContainerStarted","Data":"12b0e15f1b6f5398d00f3b0bd90466e90023980bbcde394fe8022a4f7ac59be7"}
Jul 07 06:39:41.315957 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:41.315468    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-g8q6w"
Jul 07 06:39:41.316903 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:41.316881    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rm2nb" event={"ID":"9e15e92a-9c42-4f51-91eb-0bf48eda4502","Type":"ContainerStarted","Data":"7b6c930891dacf17235d0e96eb65806bb83a7fc3d53a0db43124bed686766122"}
Jul 07 06:39:41.331208 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:41.331136    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-g8q6w" podStartSLOduration=1.5994047340000002 podStartE2EDuration="6.331122744s" totalImagesPullingTime="4.73171801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:39:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:39:41.330417423 +0000 UTC m=+464.694613986" watchObservedRunningTime="2026-07-07 06:39:41.331122744 +0000 UTC m=+464.695319289"
Jul 07 06:39:41.344052 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:41.344004    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-rm2nb" podStartSLOduration=1.422377061 podStartE2EDuration="5.343987017s" totalImagesPullingTime="3.921609956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:39:41.343453208 +0000 UTC m=+464.707649755" watchObservedRunningTime="2026-07-07 06:39:41.343987017 +0000 UTC m=+464.708183562"
Jul 07 06:39:41.907334 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:41.907298    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-rm2nb"]
Jul 07 06:39:43.325699 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:43.325654    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-rm2nb" podUID="9e15e92a-9c42-4f51-91eb-0bf48eda4502" containerName="authorino" containerID="cri-o://7b6c930891dacf17235d0e96eb65806bb83a7fc3d53a0db43124bed686766122" gracePeriod=30
Jul 07 06:39:43.611330 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:43.611290    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rm2nb"
Jul 07 06:39:43.730895 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:43.730855    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvdxd\" (UniqueName: \"kubernetes.io/projected/9e15e92a-9c42-4f51-91eb-0bf48eda4502-kube-api-access-tvdxd\") pod \"9e15e92a-9c42-4f51-91eb-0bf48eda4502\" (UID: \"9e15e92a-9c42-4f51-91eb-0bf48eda4502\") "
Jul 07 06:39:43.733133 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:43.733106    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e15e92a-9c42-4f51-91eb-0bf48eda4502-kube-api-access-tvdxd" (OuterVolumeSpecName: "kube-api-access-tvdxd") pod "9e15e92a-9c42-4f51-91eb-0bf48eda4502" (UID: "9e15e92a-9c42-4f51-91eb-0bf48eda4502"). InnerVolumeSpecName "kube-api-access-tvdxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 06:39:43.831986 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:43.831899    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvdxd\" (UniqueName: \"kubernetes.io/projected/9e15e92a-9c42-4f51-91eb-0bf48eda4502-kube-api-access-tvdxd\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:39:44.331544 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:44.331502    2563 generic.go:356] "Generic (PLEG): container finished" podID="9e15e92a-9c42-4f51-91eb-0bf48eda4502" containerID="7b6c930891dacf17235d0e96eb65806bb83a7fc3d53a0db43124bed686766122" exitCode=0
Jul 07 06:39:44.332036 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:44.331557    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rm2nb"
Jul 07 06:39:44.332036 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:44.331590    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rm2nb" event={"ID":"9e15e92a-9c42-4f51-91eb-0bf48eda4502","Type":"ContainerDied","Data":"7b6c930891dacf17235d0e96eb65806bb83a7fc3d53a0db43124bed686766122"}
Jul 07 06:39:44.332036 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:44.331656    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rm2nb" event={"ID":"9e15e92a-9c42-4f51-91eb-0bf48eda4502","Type":"ContainerDied","Data":"d81822b57333b92974928668dc231bd1f2cc533120d8c83f9da7779373ec30a9"}
Jul 07 06:39:44.332036 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:44.331674    2563 scope.go:117] "RemoveContainer" containerID="7b6c930891dacf17235d0e96eb65806bb83a7fc3d53a0db43124bed686766122"
Jul 07 06:39:44.340324 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:44.340302    2563 scope.go:117] "RemoveContainer" containerID="7b6c930891dacf17235d0e96eb65806bb83a7fc3d53a0db43124bed686766122"
Jul 07 06:39:44.340612 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:39:44.340594    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b6c930891dacf17235d0e96eb65806bb83a7fc3d53a0db43124bed686766122\": container with ID starting with 7b6c930891dacf17235d0e96eb65806bb83a7fc3d53a0db43124bed686766122 not found: ID does not exist" containerID="7b6c930891dacf17235d0e96eb65806bb83a7fc3d53a0db43124bed686766122"
Jul 07 06:39:44.340683 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:44.340623    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b6c930891dacf17235d0e96eb65806bb83a7fc3d53a0db43124bed686766122"} err="failed to get container status \"7b6c930891dacf17235d0e96eb65806bb83a7fc3d53a0db43124bed686766122\": rpc error: code = NotFound desc = could not find container \"7b6c930891dacf17235d0e96eb65806bb83a7fc3d53a0db43124bed686766122\": container with ID starting with 7b6c930891dacf17235d0e96eb65806bb83a7fc3d53a0db43124bed686766122 not found: ID does not exist"
Jul 07 06:39:44.350820 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:44.350790    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-rm2nb"]
Jul 07 06:39:44.354169 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:44.354144    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-rm2nb"]
Jul 07 06:39:45.171790 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:45.171749    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e15e92a-9c42-4f51-91eb-0bf48eda4502" path="/var/lib/kubelet/pods/9e15e92a-9c42-4f51-91eb-0bf48eda4502/volumes"
Jul 07 06:39:52.322296 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:52.322264    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-g8q6w"
Jul 07 06:39:52.920820 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:52.920789    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-g8q6w"]
Jul 07 06:39:52.921020 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:52.920961    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-g8q6w" podUID="c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d" containerName="limitador" containerID="cri-o://12b0e15f1b6f5398d00f3b0bd90466e90023980bbcde394fe8022a4f7ac59be7" gracePeriod=30
Jul 07 06:39:53.374709 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:53.374677    2563 generic.go:356] "Generic (PLEG): container finished" podID="c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d" containerID="12b0e15f1b6f5398d00f3b0bd90466e90023980bbcde394fe8022a4f7ac59be7" exitCode=0
Jul 07 06:39:53.375090 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:53.374725    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-g8q6w" event={"ID":"c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d","Type":"ContainerDied","Data":"12b0e15f1b6f5398d00f3b0bd90466e90023980bbcde394fe8022a4f7ac59be7"}
Jul 07 06:39:53.470507 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:53.470480    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-g8q6w"
Jul 07 06:39:53.617895 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:53.617863    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w52vc\" (UniqueName: \"kubernetes.io/projected/c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d-kube-api-access-w52vc\") pod \"c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d\" (UID: \"c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d\") "
Jul 07 06:39:53.618070 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:53.617994    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d-config-file\") pod \"c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d\" (UID: \"c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d\") "
Jul 07 06:39:53.618395 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:53.618364    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d-config-file" (OuterVolumeSpecName: "config-file") pod "c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d" (UID: "c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 06:39:53.620261 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:53.620234    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d-kube-api-access-w52vc" (OuterVolumeSpecName: "kube-api-access-w52vc") pod "c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d" (UID: "c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d"). InnerVolumeSpecName "kube-api-access-w52vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 06:39:53.719530 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:53.719487    2563 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d-config-file\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:39:53.719530 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:53.719524    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w52vc\" (UniqueName: \"kubernetes.io/projected/c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d-kube-api-access-w52vc\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:39:54.379743 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:54.379706    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-g8q6w" event={"ID":"c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d","Type":"ContainerDied","Data":"99e0f67fbe2d33a98b4207da53c8554d9f6ba57097f2fbea8fcb95e18d589000"}
Jul 07 06:39:54.379743 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:54.379737    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-g8q6w"
Jul 07 06:39:54.380271 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:54.379768    2563 scope.go:117] "RemoveContainer" containerID="12b0e15f1b6f5398d00f3b0bd90466e90023980bbcde394fe8022a4f7ac59be7"
Jul 07 06:39:54.399461 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:54.399430    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-g8q6w"]
Jul 07 06:39:54.403773 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:54.403742    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-g8q6w"]
Jul 07 06:39:54.624825 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:54.624788    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-b6stw"]
Jul 07 06:39:54.625111 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:54.625098    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d" containerName="limitador"
Jul 07 06:39:54.625168 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:54.625113    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d" containerName="limitador"
Jul 07 06:39:54.625168 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:54.625134    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e15e92a-9c42-4f51-91eb-0bf48eda4502" containerName="authorino"
Jul 07 06:39:54.625168 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:54.625140    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e15e92a-9c42-4f51-91eb-0bf48eda4502" containerName="authorino"
Jul 07 06:39:54.625257 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:54.625186    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d" containerName="limitador"
Jul 07 06:39:54.625257 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:54.625195    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e15e92a-9c42-4f51-91eb-0bf48eda4502" containerName="authorino"
Jul 07 06:39:54.633377 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:54.633309    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-b6stw"
Jul 07 06:39:54.635475 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:54.635443    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-b6stw"]
Jul 07 06:39:54.635660 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:54.635614    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\""
Jul 07 06:39:54.635799 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:54.635776    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gr682\""
Jul 07 06:39:54.727686 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:54.727614    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0d222fa8-990f-4c76-b79a-abbca96a5cdd-data\") pod \"postgres-69fb4594cf-b6stw\" (UID: \"0d222fa8-990f-4c76-b79a-abbca96a5cdd\") " pod="opendatahub/postgres-69fb4594cf-b6stw"
Jul 07 06:39:54.727873 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:54.727746    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6pds\" (UniqueName: \"kubernetes.io/projected/0d222fa8-990f-4c76-b79a-abbca96a5cdd-kube-api-access-p6pds\") pod \"postgres-69fb4594cf-b6stw\" (UID: \"0d222fa8-990f-4c76-b79a-abbca96a5cdd\") " pod="opendatahub/postgres-69fb4594cf-b6stw"
Jul 07 06:39:54.829036 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:54.828988    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0d222fa8-990f-4c76-b79a-abbca96a5cdd-data\") pod \"postgres-69fb4594cf-b6stw\" (UID: \"0d222fa8-990f-4c76-b79a-abbca96a5cdd\") " pod="opendatahub/postgres-69fb4594cf-b6stw"
Jul 07 06:39:54.829214 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:54.829054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6pds\" (UniqueName: \"kubernetes.io/projected/0d222fa8-990f-4c76-b79a-abbca96a5cdd-kube-api-access-p6pds\") pod \"postgres-69fb4594cf-b6stw\" (UID: \"0d222fa8-990f-4c76-b79a-abbca96a5cdd\") " pod="opendatahub/postgres-69fb4594cf-b6stw"
Jul 07 06:39:54.829389 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:54.829368    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0d222fa8-990f-4c76-b79a-abbca96a5cdd-data\") pod \"postgres-69fb4594cf-b6stw\" (UID: \"0d222fa8-990f-4c76-b79a-abbca96a5cdd\") " pod="opendatahub/postgres-69fb4594cf-b6stw"
Jul 07 06:39:54.838090 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:54.838060    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6pds\" (UniqueName: \"kubernetes.io/projected/0d222fa8-990f-4c76-b79a-abbca96a5cdd-kube-api-access-p6pds\") pod \"postgres-69fb4594cf-b6stw\" (UID: \"0d222fa8-990f-4c76-b79a-abbca96a5cdd\") " pod="opendatahub/postgres-69fb4594cf-b6stw"
Jul 07 06:39:54.945190 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:54.945079    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-b6stw"
Jul 07 06:39:55.085339 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:55.085207    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-b6stw"]
Jul 07 06:39:55.088180 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:39:55.088147    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d222fa8_990f_4c76_b79a_abbca96a5cdd.slice/crio-349742f1743786c9ef6aa5d35517f10b3c169ae7bf1d761340194932cfdb1659 WatchSource:0}: Error finding container 349742f1743786c9ef6aa5d35517f10b3c169ae7bf1d761340194932cfdb1659: Status 404 returned error can't find the container with id 349742f1743786c9ef6aa5d35517f10b3c169ae7bf1d761340194932cfdb1659
Jul 07 06:39:55.171798 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:55.171760    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d" path="/var/lib/kubelet/pods/c4d33a7f-f3f3-4c2b-b88c-55448ebecf2d/volumes"
Jul 07 06:39:55.391545 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:55.391503    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-b6stw" event={"ID":"0d222fa8-990f-4c76-b79a-abbca96a5cdd","Type":"ContainerStarted","Data":"349742f1743786c9ef6aa5d35517f10b3c169ae7bf1d761340194932cfdb1659"}
Jul 07 06:39:59.795974 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:39:59.795953    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\""
Jul 07 06:40:00.419300 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:00.419263    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-b6stw" event={"ID":"0d222fa8-990f-4c76-b79a-abbca96a5cdd","Type":"ContainerStarted","Data":"457452e14218fb6fa16e064d235d78770527615b05d012c8b1bb899e2664a2db"}
Jul 07 06:40:00.419471 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:00.419361    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-b6stw"
Jul 07 06:40:00.435881 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:00.435830    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-b6stw" podStartSLOduration=1.7318315530000001 podStartE2EDuration="6.435813699s" totalImagesPullingTime="4.703982146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:40:00.434746616 +0000 UTC m=+483.798943178" watchObservedRunningTime="2026-07-07 06:40:00.435813699 +0000 UTC m=+483.800010244"
Jul 07 06:40:06.456802 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:06.456774    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-b6stw"
Jul 07 06:40:06.995945 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:06.995910    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-std9q"]
Jul 07 06:40:07.002329 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.002306    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-std9q"
Jul 07 06:40:07.004865 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.004842    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-k296v\""
Jul 07 06:40:07.008067 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.008042    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-std9q"]
Jul 07 06:40:07.144252 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.144204    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc6mx\" (UniqueName: \"kubernetes.io/projected/b514aaa9-decb-4160-8d66-b5ef9eb72421-kube-api-access-tc6mx\") pod \"authorino-5d4b67c446-std9q\" (UID: \"b514aaa9-decb-4160-8d66-b5ef9eb72421\") " pod="kuadrant-system/authorino-5d4b67c446-std9q"
Jul 07 06:40:07.235674 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.235619    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-std9q"]
Jul 07 06:40:07.235948 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:40:07.235927    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-tc6mx], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-std9q" podUID="b514aaa9-decb-4160-8d66-b5ef9eb72421"
Jul 07 06:40:07.245231 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.245193    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc6mx\" (UniqueName: \"kubernetes.io/projected/b514aaa9-decb-4160-8d66-b5ef9eb72421-kube-api-access-tc6mx\") pod \"authorino-5d4b67c446-std9q\" (UID: \"b514aaa9-decb-4160-8d66-b5ef9eb72421\") " pod="kuadrant-system/authorino-5d4b67c446-std9q"
Jul 07 06:40:07.256255 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.256182    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc6mx\" (UniqueName: \"kubernetes.io/projected/b514aaa9-decb-4160-8d66-b5ef9eb72421-kube-api-access-tc6mx\") pod \"authorino-5d4b67c446-std9q\" (UID: \"b514aaa9-decb-4160-8d66-b5ef9eb72421\") " pod="kuadrant-system/authorino-5d4b67c446-std9q"
Jul 07 06:40:07.273673 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.273624    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6c669d9496-8w82d"]
Jul 07 06:40:07.277549 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.277528    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c669d9496-8w82d"
Jul 07 06:40:07.294284 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.294255    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c669d9496-8w82d"]
Jul 07 06:40:07.383734 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.383696    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c669d9496-8w82d"]
Jul 07 06:40:07.383953 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:40:07.383934    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-4jhh8], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6c669d9496-8w82d" podUID="50e4e202-eccd-40bd-bbf0-20fc30d65798"
Jul 07 06:40:07.409077 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.409043    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58848b7fb8-qj27w"]
Jul 07 06:40:07.413243 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.413223    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58848b7fb8-qj27w"
Jul 07 06:40:07.415449 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.415422    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 07 06:40:07.420796 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.420769    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58848b7fb8-qj27w"]
Jul 07 06:40:07.447369 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.447335    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jhh8\" (UniqueName: \"kubernetes.io/projected/50e4e202-eccd-40bd-bbf0-20fc30d65798-kube-api-access-4jhh8\") pod \"authorino-6c669d9496-8w82d\" (UID: \"50e4e202-eccd-40bd-bbf0-20fc30d65798\") " pod="kuadrant-system/authorino-6c669d9496-8w82d"
Jul 07 06:40:07.457137 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.457100    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c669d9496-8w82d"
Jul 07 06:40:07.457545 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.457102    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-std9q"
Jul 07 06:40:07.464371 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.464348    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c669d9496-8w82d"
Jul 07 06:40:07.467399 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.467380    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-std9q"
Jul 07 06:40:07.548677 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.548542    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tc6mx\" (UniqueName: \"kubernetes.io/projected/b514aaa9-decb-4160-8d66-b5ef9eb72421-kube-api-access-tc6mx\") pod \"b514aaa9-decb-4160-8d66-b5ef9eb72421\" (UID: \"b514aaa9-decb-4160-8d66-b5ef9eb72421\") "
Jul 07 06:40:07.548862 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.548690    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/40a11f6d-484b-4e49-a4c6-0cc2fe85c501-tls-cert\") pod \"authorino-58848b7fb8-qj27w\" (UID: \"40a11f6d-484b-4e49-a4c6-0cc2fe85c501\") " pod="kuadrant-system/authorino-58848b7fb8-qj27w"
Jul 07 06:40:07.548862 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.548749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jhh8\" (UniqueName: \"kubernetes.io/projected/50e4e202-eccd-40bd-bbf0-20fc30d65798-kube-api-access-4jhh8\") pod \"authorino-6c669d9496-8w82d\" (UID: \"50e4e202-eccd-40bd-bbf0-20fc30d65798\") " pod="kuadrant-system/authorino-6c669d9496-8w82d"
Jul 07 06:40:07.548862 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.548795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs4jk\" (UniqueName: \"kubernetes.io/projected/40a11f6d-484b-4e49-a4c6-0cc2fe85c501-kube-api-access-bs4jk\") pod \"authorino-58848b7fb8-qj27w\" (UID: \"40a11f6d-484b-4e49-a4c6-0cc2fe85c501\") " pod="kuadrant-system/authorino-58848b7fb8-qj27w"
Jul 07 06:40:07.550879 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.550850    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b514aaa9-decb-4160-8d66-b5ef9eb72421-kube-api-access-tc6mx" (OuterVolumeSpecName: "kube-api-access-tc6mx") pod "b514aaa9-decb-4160-8d66-b5ef9eb72421" (UID: "b514aaa9-decb-4160-8d66-b5ef9eb72421"). InnerVolumeSpecName "kube-api-access-tc6mx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 06:40:07.557587 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.557557    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jhh8\" (UniqueName: \"kubernetes.io/projected/50e4e202-eccd-40bd-bbf0-20fc30d65798-kube-api-access-4jhh8\") pod \"authorino-6c669d9496-8w82d\" (UID: \"50e4e202-eccd-40bd-bbf0-20fc30d65798\") " pod="kuadrant-system/authorino-6c669d9496-8w82d"
Jul 07 06:40:07.649439 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.649416    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jhh8\" (UniqueName: \"kubernetes.io/projected/50e4e202-eccd-40bd-bbf0-20fc30d65798-kube-api-access-4jhh8\") pod \"50e4e202-eccd-40bd-bbf0-20fc30d65798\" (UID: \"50e4e202-eccd-40bd-bbf0-20fc30d65798\") "
Jul 07 06:40:07.649556 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.649514    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/40a11f6d-484b-4e49-a4c6-0cc2fe85c501-tls-cert\") pod \"authorino-58848b7fb8-qj27w\" (UID: \"40a11f6d-484b-4e49-a4c6-0cc2fe85c501\") " pod="kuadrant-system/authorino-58848b7fb8-qj27w"
Jul 07 06:40:07.649593 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.649569    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs4jk\" (UniqueName: \"kubernetes.io/projected/40a11f6d-484b-4e49-a4c6-0cc2fe85c501-kube-api-access-bs4jk\") pod \"authorino-58848b7fb8-qj27w\" (UID: \"40a11f6d-484b-4e49-a4c6-0cc2fe85c501\") " pod="kuadrant-system/authorino-58848b7fb8-qj27w"
Jul 07 06:40:07.649630 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.649605    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc6mx\" (UniqueName: \"kubernetes.io/projected/b514aaa9-decb-4160-8d66-b5ef9eb72421-kube-api-access-tc6mx\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:40:07.651672 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.651609    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50e4e202-eccd-40bd-bbf0-20fc30d65798-kube-api-access-4jhh8" (OuterVolumeSpecName: "kube-api-access-4jhh8") pod "50e4e202-eccd-40bd-bbf0-20fc30d65798" (UID: "50e4e202-eccd-40bd-bbf0-20fc30d65798"). InnerVolumeSpecName "kube-api-access-4jhh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 06:40:07.652047 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.652028    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/40a11f6d-484b-4e49-a4c6-0cc2fe85c501-tls-cert\") pod \"authorino-58848b7fb8-qj27w\" (UID: \"40a11f6d-484b-4e49-a4c6-0cc2fe85c501\") " pod="kuadrant-system/authorino-58848b7fb8-qj27w"
Jul 07 06:40:07.658154 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.658129    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs4jk\" (UniqueName: \"kubernetes.io/projected/40a11f6d-484b-4e49-a4c6-0cc2fe85c501-kube-api-access-bs4jk\") pod \"authorino-58848b7fb8-qj27w\" (UID: \"40a11f6d-484b-4e49-a4c6-0cc2fe85c501\") " pod="kuadrant-system/authorino-58848b7fb8-qj27w"
Jul 07 06:40:07.723951 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.723897    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58848b7fb8-qj27w"
Jul 07 06:40:07.750225 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.750193    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jhh8\" (UniqueName: \"kubernetes.io/projected/50e4e202-eccd-40bd-bbf0-20fc30d65798-kube-api-access-4jhh8\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:40:07.859926 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:07.859802    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58848b7fb8-qj27w"]
Jul 07 06:40:07.862539 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:40:07.862509    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40a11f6d_484b_4e49_a4c6_0cc2fe85c501.slice/crio-5c8ac8fa2a0b45734c52fbf5b7efbe509d25dc94ac0bb8d0381e38288ef1b218 WatchSource:0}: Error finding container 5c8ac8fa2a0b45734c52fbf5b7efbe509d25dc94ac0bb8d0381e38288ef1b218: Status 404 returned error can't find the container with id 5c8ac8fa2a0b45734c52fbf5b7efbe509d25dc94ac0bb8d0381e38288ef1b218
Jul 07 06:40:08.463190 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:08.463154    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-std9q"
Jul 07 06:40:08.463591 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:08.463189    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58848b7fb8-qj27w" event={"ID":"40a11f6d-484b-4e49-a4c6-0cc2fe85c501","Type":"ContainerStarted","Data":"16d1b8fed429a62f90e3c33b9033ac7521005a7b7f3a2347cc9eac6e63deae82"}
Jul 07 06:40:08.463591 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:08.463235    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58848b7fb8-qj27w" event={"ID":"40a11f6d-484b-4e49-a4c6-0cc2fe85c501","Type":"ContainerStarted","Data":"5c8ac8fa2a0b45734c52fbf5b7efbe509d25dc94ac0bb8d0381e38288ef1b218"}
Jul 07 06:40:08.463591 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:08.463327    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c669d9496-8w82d"
Jul 07 06:40:08.481861 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:08.481814    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58848b7fb8-qj27w" podStartSLOduration=1.062078209 podStartE2EDuration="1.481800261s" totalImagesPullingTime="419.722052ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:40:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:40:08.481017277 +0000 UTC m=+491.845213836" watchObservedRunningTime="2026-07-07 06:40:08.481800261 +0000 UTC m=+491.845996806"
Jul 07 06:40:08.505674 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:08.505629    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c669d9496-8w82d"]
Jul 07 06:40:08.514780 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:08.514749    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6c669d9496-8w82d"]
Jul 07 06:40:08.538540 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:08.538464    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-std9q"]
Jul 07 06:40:08.545347 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:08.545319    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-std9q"]
Jul 07 06:40:09.171448 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:09.171414    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50e4e202-eccd-40bd-bbf0-20fc30d65798" path="/var/lib/kubelet/pods/50e4e202-eccd-40bd-bbf0-20fc30d65798/volumes"
Jul 07 06:40:09.171627 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:09.171616    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b514aaa9-decb-4160-8d66-b5ef9eb72421" path="/var/lib/kubelet/pods/b514aaa9-decb-4160-8d66-b5ef9eb72421/volumes"
Jul 07 06:40:13.144801 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:13.144763    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77b55b5c58-8nt6m"]
Jul 07 06:40:13.302762 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:13.302723    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77b55b5c58-8nt6m"
Jul 07 06:40:13.305103 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:13.305082    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-scvt9\""
Jul 07 06:40:13.305247 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:13.305081    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 07 06:40:13.305247 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:13.305123    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 07 06:40:13.308404 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:13.308383    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77b55b5c58-8nt6m"]
Jul 07 06:40:13.500370 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:13.500292    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj2v6\" (UniqueName: \"kubernetes.io/projected/26bb19c2-b093-46af-9ba0-62d0b86a49c7-kube-api-access-tj2v6\") pod \"maas-controller-77b55b5c58-8nt6m\" (UID: \"26bb19c2-b093-46af-9ba0-62d0b86a49c7\") " pod="opendatahub/maas-controller-77b55b5c58-8nt6m"
Jul 07 06:40:13.500370 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:13.500336    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/26bb19c2-b093-46af-9ba0-62d0b86a49c7-webhook-cert\") pod \"maas-controller-77b55b5c58-8nt6m\" (UID: \"26bb19c2-b093-46af-9ba0-62d0b86a49c7\") " pod="opendatahub/maas-controller-77b55b5c58-8nt6m"
Jul 07 06:40:13.601599 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:13.601562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj2v6\" (UniqueName: \"kubernetes.io/projected/26bb19c2-b093-46af-9ba0-62d0b86a49c7-kube-api-access-tj2v6\") pod \"maas-controller-77b55b5c58-8nt6m\" (UID: \"26bb19c2-b093-46af-9ba0-62d0b86a49c7\") " pod="opendatahub/maas-controller-77b55b5c58-8nt6m"
Jul 07 06:40:13.601793 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:13.601613    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/26bb19c2-b093-46af-9ba0-62d0b86a49c7-webhook-cert\") pod \"maas-controller-77b55b5c58-8nt6m\" (UID: \"26bb19c2-b093-46af-9ba0-62d0b86a49c7\") " pod="opendatahub/maas-controller-77b55b5c58-8nt6m"
Jul 07 06:40:13.601793 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:40:13.601784    2563 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 07 06:40:13.601864 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:40:13.601837    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26bb19c2-b093-46af-9ba0-62d0b86a49c7-webhook-cert podName:26bb19c2-b093-46af-9ba0-62d0b86a49c7 nodeName:}" failed. No retries permitted until 2026-07-07 06:40:14.101818332 +0000 UTC m=+497.466014855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/26bb19c2-b093-46af-9ba0-62d0b86a49c7-webhook-cert") pod "maas-controller-77b55b5c58-8nt6m" (UID: "26bb19c2-b093-46af-9ba0-62d0b86a49c7") : secret "maas-controller-webhook-cert" not found
Jul 07 06:40:13.611065 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:13.611034    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj2v6\" (UniqueName: \"kubernetes.io/projected/26bb19c2-b093-46af-9ba0-62d0b86a49c7-kube-api-access-tj2v6\") pod \"maas-controller-77b55b5c58-8nt6m\" (UID: \"26bb19c2-b093-46af-9ba0-62d0b86a49c7\") " pod="opendatahub/maas-controller-77b55b5c58-8nt6m"
Jul 07 06:40:14.106163 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:14.106126    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/26bb19c2-b093-46af-9ba0-62d0b86a49c7-webhook-cert\") pod \"maas-controller-77b55b5c58-8nt6m\" (UID: \"26bb19c2-b093-46af-9ba0-62d0b86a49c7\") " pod="opendatahub/maas-controller-77b55b5c58-8nt6m"
Jul 07 06:40:14.108561 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:14.108535    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/26bb19c2-b093-46af-9ba0-62d0b86a49c7-webhook-cert\") pod \"maas-controller-77b55b5c58-8nt6m\" (UID: \"26bb19c2-b093-46af-9ba0-62d0b86a49c7\") " pod="opendatahub/maas-controller-77b55b5c58-8nt6m"
Jul 07 06:40:14.212930 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:14.212889    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77b55b5c58-8nt6m"
Jul 07 06:40:14.405050 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:14.405024    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77b55b5c58-8nt6m"]
Jul 07 06:40:14.407513 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:40:14.407488    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26bb19c2_b093_46af_9ba0_62d0b86a49c7.slice/crio-c26b28dc211b7f301f83c9efd245514d5502635e987f57a608a756e7a96cce2c WatchSource:0}: Error finding container c26b28dc211b7f301f83c9efd245514d5502635e987f57a608a756e7a96cce2c: Status 404 returned error can't find the container with id c26b28dc211b7f301f83c9efd245514d5502635e987f57a608a756e7a96cce2c
Jul 07 06:40:14.489326 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:14.489293    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77b55b5c58-8nt6m" event={"ID":"26bb19c2-b093-46af-9ba0-62d0b86a49c7","Type":"ContainerStarted","Data":"c26b28dc211b7f301f83c9efd245514d5502635e987f57a608a756e7a96cce2c"}
Jul 07 06:40:17.506091 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:17.506054    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77b55b5c58-8nt6m" event={"ID":"26bb19c2-b093-46af-9ba0-62d0b86a49c7","Type":"ContainerStarted","Data":"8d4f6d345142e19bcbf7d2113eebb98728a481729442603c5d43496bc21de3bd"}
Jul 07 06:40:17.506578 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:17.506156    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77b55b5c58-8nt6m"
Jul 07 06:40:17.527924 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:17.527873    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-77b55b5c58-8nt6m" podStartSLOduration=2.383290939 podStartE2EDuration="4.527855645s" totalImagesPullingTime="2.144564706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:40:17.526233317 +0000 UTC m=+500.890429892" watchObservedRunningTime="2026-07-07 06:40:17.527855645 +0000 UTC m=+500.892052205"
Jul 07 06:40:28.515456 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:28.515428    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77b55b5c58-8nt6m"
Jul 07 06:40:49.130790 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:49.130715    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-rklkm"]
Jul 07 06:40:49.133940 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:49.133915    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-rklkm"
Jul 07 06:40:49.136437 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:49.136417    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-4g2pv\""
Jul 07 06:40:49.136871 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:49.136851    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 07 06:40:49.144350 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:49.144328    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-lk6nm"]
Jul 07 06:40:49.147889 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:49.147872    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-rklkm"]
Jul 07 06:40:49.147967 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:49.147952    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-lk6nm"
Jul 07 06:40:49.162587 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:49.162562    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-lk6nm"]
Jul 07 06:40:49.219856 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:49.219827    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be371c12-0c30-4424-8f5d-ac6a5883f509-plugins-config-volume\") pod \"payload-pre-processing-7964457696-rklkm\" (UID: \"be371c12-0c30-4424-8f5d-ac6a5883f509\") " pod="openshift-ingress/payload-pre-processing-7964457696-rklkm"
Jul 07 06:40:49.219856 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:49.219860    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9qht\" (UniqueName: \"kubernetes.io/projected/be371c12-0c30-4424-8f5d-ac6a5883f509-kube-api-access-c9qht\") pod \"payload-pre-processing-7964457696-rklkm\" (UID: \"be371c12-0c30-4424-8f5d-ac6a5883f509\") " pod="openshift-ingress/payload-pre-processing-7964457696-rklkm"
Jul 07 06:40:49.321181 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:49.321142    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be371c12-0c30-4424-8f5d-ac6a5883f509-plugins-config-volume\") pod \"payload-pre-processing-7964457696-rklkm\" (UID: \"be371c12-0c30-4424-8f5d-ac6a5883f509\") " pod="openshift-ingress/payload-pre-processing-7964457696-rklkm"
Jul 07 06:40:49.321181 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:49.321182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28572129-d52e-4589-9993-2adf95c458e2-plugins-config-volume\") pod \"payload-processing-58f86c7798-lk6nm\" (UID: \"28572129-d52e-4589-9993-2adf95c458e2\") " pod="openshift-ingress/payload-processing-58f86c7798-lk6nm"
Jul 07 06:40:49.321427 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:49.321203    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9qht\" (UniqueName: \"kubernetes.io/projected/be371c12-0c30-4424-8f5d-ac6a5883f509-kube-api-access-c9qht\") pod \"payload-pre-processing-7964457696-rklkm\" (UID: \"be371c12-0c30-4424-8f5d-ac6a5883f509\") " pod="openshift-ingress/payload-pre-processing-7964457696-rklkm"
Jul 07 06:40:49.321427 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:49.321307    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p27q\" (UniqueName: \"kubernetes.io/projected/28572129-d52e-4589-9993-2adf95c458e2-kube-api-access-7p27q\") pod \"payload-processing-58f86c7798-lk6nm\" (UID: \"28572129-d52e-4589-9993-2adf95c458e2\") " pod="openshift-ingress/payload-processing-58f86c7798-lk6nm"
Jul 07 06:40:49.321902 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:49.321886    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be371c12-0c30-4424-8f5d-ac6a5883f509-plugins-config-volume\") pod \"payload-pre-processing-7964457696-rklkm\" (UID: \"be371c12-0c30-4424-8f5d-ac6a5883f509\") " pod="openshift-ingress/payload-pre-processing-7964457696-rklkm"
Jul 07 06:40:49.329341 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:49.329320    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9qht\" (UniqueName: \"kubernetes.io/projected/be371c12-0c30-4424-8f5d-ac6a5883f509-kube-api-access-c9qht\") pod \"payload-pre-processing-7964457696-rklkm\" (UID: \"be371c12-0c30-4424-8f5d-ac6a5883f509\") " pod="openshift-ingress/payload-pre-processing-7964457696-rklkm"
Jul 07 06:40:49.422655 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:49.422557    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p27q\" (UniqueName: \"kubernetes.io/projected/28572129-d52e-4589-9993-2adf95c458e2-kube-api-access-7p27q\") pod \"payload-processing-58f86c7798-lk6nm\" (UID: \"28572129-d52e-4589-9993-2adf95c458e2\") " pod="openshift-ingress/payload-processing-58f86c7798-lk6nm"
Jul 07 06:40:49.422655 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:49.422621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28572129-d52e-4589-9993-2adf95c458e2-plugins-config-volume\") pod \"payload-processing-58f86c7798-lk6nm\" (UID: \"28572129-d52e-4589-9993-2adf95c458e2\") " pod="openshift-ingress/payload-processing-58f86c7798-lk6nm"
Jul 07 06:40:49.423243 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:49.423226    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28572129-d52e-4589-9993-2adf95c458e2-plugins-config-volume\") pod \"payload-processing-58f86c7798-lk6nm\" (UID: \"28572129-d52e-4589-9993-2adf95c458e2\") " pod="openshift-ingress/payload-processing-58f86c7798-lk6nm"
Jul 07 06:40:49.431859 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:49.431833    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p27q\" (UniqueName: \"kubernetes.io/projected/28572129-d52e-4589-9993-2adf95c458e2-kube-api-access-7p27q\") pod \"payload-processing-58f86c7798-lk6nm\" (UID: \"28572129-d52e-4589-9993-2adf95c458e2\") " pod="openshift-ingress/payload-processing-58f86c7798-lk6nm"
Jul 07 06:40:49.446226 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:49.446202    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-rklkm"
Jul 07 06:40:49.457221 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:49.457201    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-lk6nm"
Jul 07 06:40:49.607252 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:40:49.607217    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe371c12_0c30_4424_8f5d_ac6a5883f509.slice/crio-a4d570bb03c428d32fd42966a36f58b667f157039d2ae691d6ea4509d944e1e3 WatchSource:0}: Error finding container a4d570bb03c428d32fd42966a36f58b667f157039d2ae691d6ea4509d944e1e3: Status 404 returned error can't find the container with id a4d570bb03c428d32fd42966a36f58b667f157039d2ae691d6ea4509d944e1e3
Jul 07 06:40:49.607450 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:49.607274    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-rklkm"]
Jul 07 06:40:49.627354 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:49.627332    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-lk6nm"]
Jul 07 06:40:49.629207 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:40:49.629184    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28572129_d52e_4589_9993_2adf95c458e2.slice/crio-bb82d8a5830ea1a1fe147e79b1f325714e0b78efe3f2fb2dd70495791648b65f WatchSource:0}: Error finding container bb82d8a5830ea1a1fe147e79b1f325714e0b78efe3f2fb2dd70495791648b65f: Status 404 returned error can't find the container with id bb82d8a5830ea1a1fe147e79b1f325714e0b78efe3f2fb2dd70495791648b65f
Jul 07 06:40:49.650249 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:49.650214    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-rklkm" event={"ID":"be371c12-0c30-4424-8f5d-ac6a5883f509","Type":"ContainerStarted","Data":"a4d570bb03c428d32fd42966a36f58b667f157039d2ae691d6ea4509d944e1e3"}
Jul 07 06:40:49.651408 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:49.651383    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-lk6nm" event={"ID":"28572129-d52e-4589-9993-2adf95c458e2","Type":"ContainerStarted","Data":"bb82d8a5830ea1a1fe147e79b1f325714e0b78efe3f2fb2dd70495791648b65f"}
Jul 07 06:40:51.662540 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:51.662494    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-lk6nm" event={"ID":"28572129-d52e-4589-9993-2adf95c458e2","Type":"ContainerStarted","Data":"01f0562045e7945dbef27f6548c76b4a994775046c8988cee6f64c0fb706f561"}
Jul 07 06:40:51.662995 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:51.662664    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-lk6nm"
Jul 07 06:40:51.663931 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:51.663907    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-rklkm" event={"ID":"be371c12-0c30-4424-8f5d-ac6a5883f509","Type":"ContainerStarted","Data":"314e2d3219549294c59f6220d65ea278352f908ba46fbd45b8bf03e2a87bec3b"}
Jul 07 06:40:51.664100 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:51.664036    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-rklkm"
Jul 07 06:40:51.677248 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:51.677193    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-lk6nm" podStartSLOduration=1.301980585 podStartE2EDuration="2.67717247s" totalImagesPullingTime="1.375191885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:40:51.676215599 +0000 UTC m=+535.040412143" watchObservedRunningTime="2026-07-07 06:40:51.67717247 +0000 UTC m=+535.041369014"
Jul 07 06:40:51.695657 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:40:51.694851    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-rklkm" podStartSLOduration=1.3230285849999999 podStartE2EDuration="2.694831918s" totalImagesPullingTime="1.371803333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:40:51.690808762 +0000 UTC m=+535.055005333" watchObservedRunningTime="2026-07-07 06:40:51.694831918 +0000 UTC m=+535.059028467"
Jul 07 06:41:02.670246 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:02.670213    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-rklkm"
Jul 07 06:41:02.670661 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:02.670392    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-lk6nm"
Jul 07 06:41:30.275580 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.275543    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g"]
Jul 07 06:41:30.280095 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.280073    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g"
Jul 07 06:41:30.282958 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.282925    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 07 06:41:30.282958 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.282950    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7scrk\""
Jul 07 06:41:30.283470 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.283121    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 07 06:41:30.283470 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.283117    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 07 06:41:30.294866 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.294840    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g"]
Jul 07 06:41:30.406016 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.405981    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe3b8395-7fde-4c15-85d4-b7ff5769ea14-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fm56g\" (UID: \"fe3b8395-7fde-4c15-85d4-b7ff5769ea14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g"
Jul 07 06:41:30.406016 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.406030    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe3b8395-7fde-4c15-85d4-b7ff5769ea14-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fm56g\" (UID: \"fe3b8395-7fde-4c15-85d4-b7ff5769ea14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g"
Jul 07 06:41:30.406226 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.406065    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe3b8395-7fde-4c15-85d4-b7ff5769ea14-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fm56g\" (UID: \"fe3b8395-7fde-4c15-85d4-b7ff5769ea14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g"
Jul 07 06:41:30.406226 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.406092    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe3b8395-7fde-4c15-85d4-b7ff5769ea14-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fm56g\" (UID: \"fe3b8395-7fde-4c15-85d4-b7ff5769ea14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g"
Jul 07 06:41:30.406226 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.406119    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q75j2\" (UniqueName: \"kubernetes.io/projected/fe3b8395-7fde-4c15-85d4-b7ff5769ea14-kube-api-access-q75j2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fm56g\" (UID: \"fe3b8395-7fde-4c15-85d4-b7ff5769ea14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g"
Jul 07 06:41:30.406226 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.406190    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe3b8395-7fde-4c15-85d4-b7ff5769ea14-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fm56g\" (UID: \"fe3b8395-7fde-4c15-85d4-b7ff5769ea14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g"
Jul 07 06:41:30.506799 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.506751    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe3b8395-7fde-4c15-85d4-b7ff5769ea14-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fm56g\" (UID: \"fe3b8395-7fde-4c15-85d4-b7ff5769ea14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g"
Jul 07 06:41:30.506998 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.506832    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe3b8395-7fde-4c15-85d4-b7ff5769ea14-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fm56g\" (UID: \"fe3b8395-7fde-4c15-85d4-b7ff5769ea14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g"
Jul 07 06:41:30.506998 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.506874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe3b8395-7fde-4c15-85d4-b7ff5769ea14-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fm56g\" (UID: \"fe3b8395-7fde-4c15-85d4-b7ff5769ea14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g"
Jul 07 06:41:30.506998 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.506901    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe3b8395-7fde-4c15-85d4-b7ff5769ea14-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fm56g\" (UID: \"fe3b8395-7fde-4c15-85d4-b7ff5769ea14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g"
Jul 07 06:41:30.506998 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.506931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q75j2\" (UniqueName: \"kubernetes.io/projected/fe3b8395-7fde-4c15-85d4-b7ff5769ea14-kube-api-access-q75j2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fm56g\" (UID: \"fe3b8395-7fde-4c15-85d4-b7ff5769ea14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g"
Jul 07 06:41:30.506998 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.506972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe3b8395-7fde-4c15-85d4-b7ff5769ea14-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fm56g\" (UID: \"fe3b8395-7fde-4c15-85d4-b7ff5769ea14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g"
Jul 07 06:41:30.507360 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.507321    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe3b8395-7fde-4c15-85d4-b7ff5769ea14-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fm56g\" (UID: \"fe3b8395-7fde-4c15-85d4-b7ff5769ea14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g"
Jul 07 06:41:30.507488 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.507396    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe3b8395-7fde-4c15-85d4-b7ff5769ea14-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fm56g\" (UID: \"fe3b8395-7fde-4c15-85d4-b7ff5769ea14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g"
Jul 07 06:41:30.507488 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.507441    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe3b8395-7fde-4c15-85d4-b7ff5769ea14-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fm56g\" (UID: \"fe3b8395-7fde-4c15-85d4-b7ff5769ea14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g"
Jul 07 06:41:30.509272 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.509251    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe3b8395-7fde-4c15-85d4-b7ff5769ea14-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fm56g\" (UID: \"fe3b8395-7fde-4c15-85d4-b7ff5769ea14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g"
Jul 07 06:41:30.509518 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.509500    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe3b8395-7fde-4c15-85d4-b7ff5769ea14-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fm56g\" (UID: \"fe3b8395-7fde-4c15-85d4-b7ff5769ea14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g"
Jul 07 06:41:30.515251 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.515229    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q75j2\" (UniqueName: \"kubernetes.io/projected/fe3b8395-7fde-4c15-85d4-b7ff5769ea14-kube-api-access-q75j2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-fm56g\" (UID: \"fe3b8395-7fde-4c15-85d4-b7ff5769ea14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g"
Jul 07 06:41:30.592786 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.592753    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g"
Jul 07 06:41:30.729439 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.729411    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g"]
Jul 07 06:41:30.731519 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:41:30.731477    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe3b8395_7fde_4c15_85d4_b7ff5769ea14.slice/crio-1d426e9f4bde435ab67884ca15feaf8b36e7d1842bf6f3623373fc6c646142cd WatchSource:0}: Error finding container 1d426e9f4bde435ab67884ca15feaf8b36e7d1842bf6f3623373fc6c646142cd: Status 404 returned error can't find the container with id 1d426e9f4bde435ab67884ca15feaf8b36e7d1842bf6f3623373fc6c646142cd
Jul 07 06:41:30.841277 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:30.841238    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g" event={"ID":"fe3b8395-7fde-4c15-85d4-b7ff5769ea14","Type":"ContainerStarted","Data":"1d426e9f4bde435ab67884ca15feaf8b36e7d1842bf6f3623373fc6c646142cd"}
Jul 07 06:41:32.852506 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:32.852467    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g" event={"ID":"fe3b8395-7fde-4c15-85d4-b7ff5769ea14","Type":"ContainerStarted","Data":"09bfa50aa567951cb7861a1389f9dcec004a2ac2c71c25f66a44f8bfa178aef3"}
Jul 07 06:41:39.279093 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:39.279026    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g" podStartSLOduration=8.199142908 podStartE2EDuration="9.278978446s" totalImagesPullingTime="1.079835538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:41:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:41:32.869082989 +0000 UTC m=+576.233279556" watchObservedRunningTime="2026-07-07 06:41:39.278978446 +0000 UTC m=+582.643174992"
Jul 07 06:41:39.281328 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:39.281301    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z"]
Jul 07 06:41:39.285400 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:39.285380    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z"
Jul 07 06:41:39.287437 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:39.287409    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 07 06:41:39.291573 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:39.291545    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z"]
Jul 07 06:41:39.382573 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:39.382534    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ae6c62-626a-49ad-83f8-db6844af3425-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmp7z\" (UID: \"55ae6c62-626a-49ad-83f8-db6844af3425\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z"
Jul 07 06:41:39.382573 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:39.382576    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ae6c62-626a-49ad-83f8-db6844af3425-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmp7z\" (UID: \"55ae6c62-626a-49ad-83f8-db6844af3425\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z"
Jul 07 06:41:39.382848 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:39.382613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ae6c62-626a-49ad-83f8-db6844af3425-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmp7z\" (UID: \"55ae6c62-626a-49ad-83f8-db6844af3425\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z"
Jul 07 06:41:39.382848 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:39.382692    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ae6c62-626a-49ad-83f8-db6844af3425-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmp7z\" (UID: \"55ae6c62-626a-49ad-83f8-db6844af3425\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z"
Jul 07 06:41:39.382848 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:39.382755    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ae6c62-626a-49ad-83f8-db6844af3425-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmp7z\" (UID: \"55ae6c62-626a-49ad-83f8-db6844af3425\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z"
Jul 07 06:41:39.382848 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:39.382811    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r84j9\" (UniqueName: \"kubernetes.io/projected/55ae6c62-626a-49ad-83f8-db6844af3425-kube-api-access-r84j9\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmp7z\" (UID: \"55ae6c62-626a-49ad-83f8-db6844af3425\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z"
Jul 07 06:41:39.484090 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:39.484048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ae6c62-626a-49ad-83f8-db6844af3425-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmp7z\" (UID: \"55ae6c62-626a-49ad-83f8-db6844af3425\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z"
Jul 07 06:41:39.484090 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:39.484092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ae6c62-626a-49ad-83f8-db6844af3425-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmp7z\" (UID: \"55ae6c62-626a-49ad-83f8-db6844af3425\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z"
Jul 07 06:41:39.484341 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:39.484120    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ae6c62-626a-49ad-83f8-db6844af3425-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmp7z\" (UID: \"55ae6c62-626a-49ad-83f8-db6844af3425\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z"
Jul 07 06:41:39.484341 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:39.484159    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r84j9\" (UniqueName: \"kubernetes.io/projected/55ae6c62-626a-49ad-83f8-db6844af3425-kube-api-access-r84j9\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmp7z\" (UID: \"55ae6c62-626a-49ad-83f8-db6844af3425\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z"
Jul 07 06:41:39.484341 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:39.484194    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ae6c62-626a-49ad-83f8-db6844af3425-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmp7z\" (UID: \"55ae6c62-626a-49ad-83f8-db6844af3425\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z"
Jul 07 06:41:39.484341 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:39.484215    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ae6c62-626a-49ad-83f8-db6844af3425-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmp7z\" (UID: \"55ae6c62-626a-49ad-83f8-db6844af3425\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z"
Jul 07 06:41:39.484548 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:39.484528    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ae6c62-626a-49ad-83f8-db6844af3425-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmp7z\" (UID: \"55ae6c62-626a-49ad-83f8-db6844af3425\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z"
Jul 07 06:41:39.484604 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:39.484550    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ae6c62-626a-49ad-83f8-db6844af3425-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmp7z\" (UID: \"55ae6c62-626a-49ad-83f8-db6844af3425\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z"
Jul 07 06:41:39.484683 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:39.484603    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ae6c62-626a-49ad-83f8-db6844af3425-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmp7z\" (UID: \"55ae6c62-626a-49ad-83f8-db6844af3425\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z"
Jul 07 06:41:39.486726 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:39.486703    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ae6c62-626a-49ad-83f8-db6844af3425-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmp7z\" (UID: \"55ae6c62-626a-49ad-83f8-db6844af3425\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z"
Jul 07 06:41:39.487145 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:39.487122    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ae6c62-626a-49ad-83f8-db6844af3425-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmp7z\" (UID: \"55ae6c62-626a-49ad-83f8-db6844af3425\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z"
Jul 07 06:41:39.492191 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:39.492156    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r84j9\" (UniqueName: \"kubernetes.io/projected/55ae6c62-626a-49ad-83f8-db6844af3425-kube-api-access-r84j9\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gmp7z\" (UID: \"55ae6c62-626a-49ad-83f8-db6844af3425\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z"
Jul 07 06:41:39.598359 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:39.598319    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z"
Jul 07 06:41:39.733420 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:39.733394    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z"]
Jul 07 06:41:39.736186 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:41:39.736163    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55ae6c62_626a_49ad_83f8_db6844af3425.slice/crio-7839325fb16987412ddf6312c2019eb421735b49b91b1fc93fd882e87c008682 WatchSource:0}: Error finding container 7839325fb16987412ddf6312c2019eb421735b49b91b1fc93fd882e87c008682: Status 404 returned error can't find the container with id 7839325fb16987412ddf6312c2019eb421735b49b91b1fc93fd882e87c008682
Jul 07 06:41:39.887229 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:39.887150    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z" event={"ID":"55ae6c62-626a-49ad-83f8-db6844af3425","Type":"ContainerStarted","Data":"7839325fb16987412ddf6312c2019eb421735b49b91b1fc93fd882e87c008682"}
Jul 07 06:41:40.593288 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:40.593237    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g"
Jul 07 06:41:40.593694 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:40.593366    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g"
Jul 07 06:41:40.605877 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:40.605852    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g"
Jul 07 06:41:40.893295 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:40.893189    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z" event={"ID":"55ae6c62-626a-49ad-83f8-db6844af3425","Type":"ContainerStarted","Data":"8cb9e8d0a5c4699b808672dc7fc5a5a27a95d4c6c7ffb33cb0e9b24adc88989a"}
Jul 07 06:41:40.905123 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:40.905093    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-fm56g"
Jul 07 06:41:40.920418 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:40.920364    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z" podStartSLOduration=1.7493614 podStartE2EDuration="1.920345765s" totalImagesPullingTime="170.984365ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:41:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:41:40.919492553 +0000 UTC m=+584.283689123" watchObservedRunningTime="2026-07-07 06:41:40.920345765 +0000 UTC m=+584.284542310"
Jul 07 06:41:49.490172 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.490129    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz"]
Jul 07 06:41:49.493382 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.493362    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz"
Jul 07 06:41:49.495472 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.495446    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 07 06:41:49.502679 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.502627    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz"]
Jul 07 06:41:49.577315 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.577286    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7wpg\" (UniqueName: \"kubernetes.io/projected/ca5278a0-49dd-4b23-8677-0972b7bb97f3-kube-api-access-j7wpg\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz\" (UID: \"ca5278a0-49dd-4b23-8677-0972b7bb97f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz"
Jul 07 06:41:49.577521 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.577322    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5278a0-49dd-4b23-8677-0972b7bb97f3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz\" (UID: \"ca5278a0-49dd-4b23-8677-0972b7bb97f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz"
Jul 07 06:41:49.577521 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.577348    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5278a0-49dd-4b23-8677-0972b7bb97f3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz\" (UID: \"ca5278a0-49dd-4b23-8677-0972b7bb97f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz"
Jul 07 06:41:49.577521 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.577371    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca5278a0-49dd-4b23-8677-0972b7bb97f3-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz\" (UID: \"ca5278a0-49dd-4b23-8677-0972b7bb97f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz"
Jul 07 06:41:49.577521 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.577455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5278a0-49dd-4b23-8677-0972b7bb97f3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz\" (UID: \"ca5278a0-49dd-4b23-8677-0972b7bb97f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz"
Jul 07 06:41:49.577521 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.577489    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5278a0-49dd-4b23-8677-0972b7bb97f3-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz\" (UID: \"ca5278a0-49dd-4b23-8677-0972b7bb97f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz"
Jul 07 06:41:49.599018 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.598989    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z"
Jul 07 06:41:49.599143 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.599079    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z"
Jul 07 06:41:49.611304 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.611285    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z"
Jul 07 06:41:49.678391 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.678364    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca5278a0-49dd-4b23-8677-0972b7bb97f3-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz\" (UID: \"ca5278a0-49dd-4b23-8677-0972b7bb97f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz"
Jul 07 06:41:49.678521 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.678432    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5278a0-49dd-4b23-8677-0972b7bb97f3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz\" (UID: \"ca5278a0-49dd-4b23-8677-0972b7bb97f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz"
Jul 07 06:41:49.678521 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.678460    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5278a0-49dd-4b23-8677-0972b7bb97f3-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz\" (UID: \"ca5278a0-49dd-4b23-8677-0972b7bb97f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz"
Jul 07 06:41:49.678778 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.678755    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7wpg\" (UniqueName: \"kubernetes.io/projected/ca5278a0-49dd-4b23-8677-0972b7bb97f3-kube-api-access-j7wpg\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz\" (UID: \"ca5278a0-49dd-4b23-8677-0972b7bb97f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz"
Jul 07 06:41:49.678871 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.678781    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca5278a0-49dd-4b23-8677-0972b7bb97f3-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz\" (UID: \"ca5278a0-49dd-4b23-8677-0972b7bb97f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz"
Jul 07 06:41:49.678871 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.678807    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5278a0-49dd-4b23-8677-0972b7bb97f3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz\" (UID: \"ca5278a0-49dd-4b23-8677-0972b7bb97f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz"
Jul 07 06:41:49.678871 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.678847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5278a0-49dd-4b23-8677-0972b7bb97f3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz\" (UID: \"ca5278a0-49dd-4b23-8677-0972b7bb97f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz"
Jul 07 06:41:49.679100 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.679077    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5278a0-49dd-4b23-8677-0972b7bb97f3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz\" (UID: \"ca5278a0-49dd-4b23-8677-0972b7bb97f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz"
Jul 07 06:41:49.679161 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.679117    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5278a0-49dd-4b23-8677-0972b7bb97f3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz\" (UID: \"ca5278a0-49dd-4b23-8677-0972b7bb97f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz"
Jul 07 06:41:49.680936 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.680916    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5278a0-49dd-4b23-8677-0972b7bb97f3-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz\" (UID: \"ca5278a0-49dd-4b23-8677-0972b7bb97f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz"
Jul 07 06:41:49.681214 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.681194    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5278a0-49dd-4b23-8677-0972b7bb97f3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz\" (UID: \"ca5278a0-49dd-4b23-8677-0972b7bb97f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz"
Jul 07 06:41:49.685959 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.685938    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7wpg\" (UniqueName: \"kubernetes.io/projected/ca5278a0-49dd-4b23-8677-0972b7bb97f3-kube-api-access-j7wpg\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz\" (UID: \"ca5278a0-49dd-4b23-8677-0972b7bb97f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz"
Jul 07 06:41:49.803085 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.803053    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz"
Jul 07 06:41:49.935697 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.935677    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz"]
Jul 07 06:41:49.937876 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:41:49.937849    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca5278a0_49dd_4b23_8677_0972b7bb97f3.slice/crio-ab416a7f88581b3c5a56f151b5ff082c61271afcd8a617e4bce8ab3c57e5f5f6 WatchSource:0}: Error finding container ab416a7f88581b3c5a56f151b5ff082c61271afcd8a617e4bce8ab3c57e5f5f6: Status 404 returned error can't find the container with id ab416a7f88581b3c5a56f151b5ff082c61271afcd8a617e4bce8ab3c57e5f5f6
Jul 07 06:41:49.946427 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:49.946406    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gmp7z"
Jul 07 06:41:50.940235 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:50.940156    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz" event={"ID":"ca5278a0-49dd-4b23-8677-0972b7bb97f3","Type":"ContainerStarted","Data":"a13f5682e9b30d7a3488a1de302e0fe6d9a8a4e0f093c5c0149be9edd1699425"}
Jul 07 06:41:50.940235 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:50.940201    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz" event={"ID":"ca5278a0-49dd-4b23-8677-0972b7bb97f3","Type":"ContainerStarted","Data":"ab416a7f88581b3c5a56f151b5ff082c61271afcd8a617e4bce8ab3c57e5f5f6"}
Jul 07 06:41:50.956932 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:50.956872    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz" podStartSLOduration=1.670700542 podStartE2EDuration="1.956857417s" totalImagesPullingTime="286.156875ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:41:50.956003834 +0000 UTC m=+594.320200380" watchObservedRunningTime="2026-07-07 06:41:50.956857417 +0000 UTC m=+594.321053995"
Jul 07 06:41:59.804156 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:59.804120    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz"
Jul 07 06:41:59.804531 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:59.804429    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz"
Jul 07 06:41:59.816652 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:41:59.816618    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz"
Jul 07 06:42:00.005320 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:42:00.005289    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz"
Jul 07 06:43:10.929393 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:43:10.929357    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58848b7fb8-qj27w"]
Jul 07 06:43:10.929895 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:43:10.929525    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58848b7fb8-qj27w" podUID="40a11f6d-484b-4e49-a4c6-0cc2fe85c501" containerName="authorino" containerID="cri-o://16d1b8fed429a62f90e3c33b9033ac7521005a7b7f3a2347cc9eac6e63deae82" gracePeriod=30
Jul 07 06:43:11.193689 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:43:11.193600    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58848b7fb8-qj27w"
Jul 07 06:43:11.316243 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:43:11.316211    2563 generic.go:356] "Generic (PLEG): container finished" podID="40a11f6d-484b-4e49-a4c6-0cc2fe85c501" containerID="16d1b8fed429a62f90e3c33b9033ac7521005a7b7f3a2347cc9eac6e63deae82" exitCode=0
Jul 07 06:43:11.316416 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:43:11.316257    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58848b7fb8-qj27w"
Jul 07 06:43:11.316416 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:43:11.316252    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58848b7fb8-qj27w" event={"ID":"40a11f6d-484b-4e49-a4c6-0cc2fe85c501","Type":"ContainerDied","Data":"16d1b8fed429a62f90e3c33b9033ac7521005a7b7f3a2347cc9eac6e63deae82"}
Jul 07 06:43:11.316416 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:43:11.316375    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58848b7fb8-qj27w" event={"ID":"40a11f6d-484b-4e49-a4c6-0cc2fe85c501","Type":"ContainerDied","Data":"5c8ac8fa2a0b45734c52fbf5b7efbe509d25dc94ac0bb8d0381e38288ef1b218"}
Jul 07 06:43:11.316416 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:43:11.316397    2563 scope.go:117] "RemoveContainer" containerID="16d1b8fed429a62f90e3c33b9033ac7521005a7b7f3a2347cc9eac6e63deae82"
Jul 07 06:43:11.316789 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:43:11.316768    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/40a11f6d-484b-4e49-a4c6-0cc2fe85c501-tls-cert\") pod \"40a11f6d-484b-4e49-a4c6-0cc2fe85c501\" (UID: \"40a11f6d-484b-4e49-a4c6-0cc2fe85c501\") "
Jul 07 06:43:11.316904 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:43:11.316803    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bs4jk\" (UniqueName: \"kubernetes.io/projected/40a11f6d-484b-4e49-a4c6-0cc2fe85c501-kube-api-access-bs4jk\") pod \"40a11f6d-484b-4e49-a4c6-0cc2fe85c501\" (UID: \"40a11f6d-484b-4e49-a4c6-0cc2fe85c501\") "
Jul 07 06:43:11.319173 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:43:11.319143    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a11f6d-484b-4e49-a4c6-0cc2fe85c501-kube-api-access-bs4jk" (OuterVolumeSpecName: "kube-api-access-bs4jk") pod "40a11f6d-484b-4e49-a4c6-0cc2fe85c501" (UID: "40a11f6d-484b-4e49-a4c6-0cc2fe85c501"). InnerVolumeSpecName "kube-api-access-bs4jk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 06:43:11.327594 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:43:11.327566    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40a11f6d-484b-4e49-a4c6-0cc2fe85c501-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "40a11f6d-484b-4e49-a4c6-0cc2fe85c501" (UID: "40a11f6d-484b-4e49-a4c6-0cc2fe85c501"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 06:43:11.336834 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:43:11.336814    2563 scope.go:117] "RemoveContainer" containerID="16d1b8fed429a62f90e3c33b9033ac7521005a7b7f3a2347cc9eac6e63deae82"
Jul 07 06:43:11.337161 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:43:11.337142    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d1b8fed429a62f90e3c33b9033ac7521005a7b7f3a2347cc9eac6e63deae82\": container with ID starting with 16d1b8fed429a62f90e3c33b9033ac7521005a7b7f3a2347cc9eac6e63deae82 not found: ID does not exist" containerID="16d1b8fed429a62f90e3c33b9033ac7521005a7b7f3a2347cc9eac6e63deae82"
Jul 07 06:43:11.337211 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:43:11.337188    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d1b8fed429a62f90e3c33b9033ac7521005a7b7f3a2347cc9eac6e63deae82"} err="failed to get container status \"16d1b8fed429a62f90e3c33b9033ac7521005a7b7f3a2347cc9eac6e63deae82\": rpc error: code = NotFound desc = could not find container \"16d1b8fed429a62f90e3c33b9033ac7521005a7b7f3a2347cc9eac6e63deae82\": container with ID starting with 16d1b8fed429a62f90e3c33b9033ac7521005a7b7f3a2347cc9eac6e63deae82 not found: ID does not exist"
Jul 07 06:43:11.423740 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:43:11.418315    2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/40a11f6d-484b-4e49-a4c6-0cc2fe85c501-tls-cert\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:43:11.423740 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:43:11.418353    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs4jk\" (UniqueName: \"kubernetes.io/projected/40a11f6d-484b-4e49-a4c6-0cc2fe85c501-kube-api-access-bs4jk\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:43:11.639837 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:43:11.639805    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58848b7fb8-qj27w"]
Jul 07 06:43:11.643773 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:43:11.643746    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58848b7fb8-qj27w"]
Jul 07 06:43:13.171690 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:43:13.171657    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40a11f6d-484b-4e49-a4c6-0cc2fe85c501" path="/var/lib/kubelet/pods/40a11f6d-484b-4e49-a4c6-0cc2fe85c501/volumes"
Jul 07 06:44:22.227942 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:44:22.227903    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77b55b5c58-8nt6m"]
Jul 07 06:44:22.228403 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:44:22.228162    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-77b55b5c58-8nt6m" podUID="26bb19c2-b093-46af-9ba0-62d0b86a49c7" containerName="manager" containerID="cri-o://8d4f6d345142e19bcbf7d2113eebb98728a481729442603c5d43496bc21de3bd" gracePeriod=30
Jul 07 06:44:22.475310 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:44:22.475282    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77b55b5c58-8nt6m"
Jul 07 06:44:22.641033 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:44:22.640996    2563 generic.go:356] "Generic (PLEG): container finished" podID="26bb19c2-b093-46af-9ba0-62d0b86a49c7" containerID="8d4f6d345142e19bcbf7d2113eebb98728a481729442603c5d43496bc21de3bd" exitCode=0
Jul 07 06:44:22.641229 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:44:22.641043    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77b55b5c58-8nt6m" event={"ID":"26bb19c2-b093-46af-9ba0-62d0b86a49c7","Type":"ContainerDied","Data":"8d4f6d345142e19bcbf7d2113eebb98728a481729442603c5d43496bc21de3bd"}
Jul 07 06:44:22.641229 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:44:22.641056    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77b55b5c58-8nt6m"
Jul 07 06:44:22.641229 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:44:22.641084    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77b55b5c58-8nt6m" event={"ID":"26bb19c2-b093-46af-9ba0-62d0b86a49c7","Type":"ContainerDied","Data":"c26b28dc211b7f301f83c9efd245514d5502635e987f57a608a756e7a96cce2c"}
Jul 07 06:44:22.641229 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:44:22.641105    2563 scope.go:117] "RemoveContainer" containerID="8d4f6d345142e19bcbf7d2113eebb98728a481729442603c5d43496bc21de3bd"
Jul 07 06:44:22.649387 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:44:22.649369    2563 scope.go:117] "RemoveContainer" containerID="8d4f6d345142e19bcbf7d2113eebb98728a481729442603c5d43496bc21de3bd"
Jul 07 06:44:22.649671 ip-10-0-133-92 kubenswrapper[2563]: E0707 06:44:22.649651    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4f6d345142e19bcbf7d2113eebb98728a481729442603c5d43496bc21de3bd\": container with ID starting with 8d4f6d345142e19bcbf7d2113eebb98728a481729442603c5d43496bc21de3bd not found: ID does not exist" containerID="8d4f6d345142e19bcbf7d2113eebb98728a481729442603c5d43496bc21de3bd"
Jul 07 06:44:22.649731 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:44:22.649682    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4f6d345142e19bcbf7d2113eebb98728a481729442603c5d43496bc21de3bd"} err="failed to get container status \"8d4f6d345142e19bcbf7d2113eebb98728a481729442603c5d43496bc21de3bd\": rpc error: code = NotFound desc = could not find container \"8d4f6d345142e19bcbf7d2113eebb98728a481729442603c5d43496bc21de3bd\": container with ID starting with 8d4f6d345142e19bcbf7d2113eebb98728a481729442603c5d43496bc21de3bd not found: ID does not exist"
Jul 07 06:44:22.665985 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:44:22.665957    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tj2v6\" (UniqueName: \"kubernetes.io/projected/26bb19c2-b093-46af-9ba0-62d0b86a49c7-kube-api-access-tj2v6\") pod \"26bb19c2-b093-46af-9ba0-62d0b86a49c7\" (UID: \"26bb19c2-b093-46af-9ba0-62d0b86a49c7\") "
Jul 07 06:44:22.666159 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:44:22.666074    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/26bb19c2-b093-46af-9ba0-62d0b86a49c7-webhook-cert\") pod \"26bb19c2-b093-46af-9ba0-62d0b86a49c7\" (UID: \"26bb19c2-b093-46af-9ba0-62d0b86a49c7\") "
Jul 07 06:44:22.668390 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:44:22.668356    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26bb19c2-b093-46af-9ba0-62d0b86a49c7-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "26bb19c2-b093-46af-9ba0-62d0b86a49c7" (UID: "26bb19c2-b093-46af-9ba0-62d0b86a49c7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 06:44:22.668508 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:44:22.668414    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26bb19c2-b093-46af-9ba0-62d0b86a49c7-kube-api-access-tj2v6" (OuterVolumeSpecName: "kube-api-access-tj2v6") pod "26bb19c2-b093-46af-9ba0-62d0b86a49c7" (UID: "26bb19c2-b093-46af-9ba0-62d0b86a49c7"). InnerVolumeSpecName "kube-api-access-tj2v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 06:44:22.766873 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:44:22.766840    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/26bb19c2-b093-46af-9ba0-62d0b86a49c7-webhook-cert\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:44:22.766873 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:44:22.766868    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj2v6\" (UniqueName: \"kubernetes.io/projected/26bb19c2-b093-46af-9ba0-62d0b86a49c7-kube-api-access-tj2v6\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 06:44:22.963972 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:44:22.963941    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77b55b5c58-8nt6m"]
Jul 07 06:44:22.970394 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:44:22.970370    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-77b55b5c58-8nt6m"]
Jul 07 06:44:23.171913 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:44:23.171872    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26bb19c2-b093-46af-9ba0-62d0b86a49c7" path="/var/lib/kubelet/pods/26bb19c2-b093-46af-9ba0-62d0b86a49c7/volumes"
Jul 07 06:59:56.093058 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.093022    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"]
Jul 07 06:59:56.093670 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.093378    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40a11f6d-484b-4e49-a4c6-0cc2fe85c501" containerName="authorino"
Jul 07 06:59:56.093670 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.093389    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="40a11f6d-484b-4e49-a4c6-0cc2fe85c501" containerName="authorino"
Jul 07 06:59:56.093670 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.093397    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26bb19c2-b093-46af-9ba0-62d0b86a49c7" containerName="manager"
Jul 07 06:59:56.093670 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.093402    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="26bb19c2-b093-46af-9ba0-62d0b86a49c7" containerName="manager"
Jul 07 06:59:56.093670 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.093454    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="40a11f6d-484b-4e49-a4c6-0cc2fe85c501" containerName="authorino"
Jul 07 06:59:56.093670 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.093461    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="26bb19c2-b093-46af-9ba0-62d0b86a49c7" containerName="manager"
Jul 07 06:59:56.096184 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.096164    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"
Jul 07 06:59:56.098557 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.098524    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f08145e7\"/\"openshift-service-ca.crt\""
Jul 07 06:59:56.098759 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.098739    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f08145e7\"/\"default-dockercfg-m7j4s\""
Jul 07 06:59:56.099115 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.099089    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f08145e7\"/\"e2e-lifecycle-model-f08145e7-kserve-self-signed-certs\""
Jul 07 06:59:56.099219 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.099152    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f08145e7\"/\"kube-root-ca.crt\""
Jul 07 06:59:56.104315 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.104289    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f75f0412-bf57-40aa-b395-1b40aedf3559-dshm\") pod \"e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79\" (UID: \"f75f0412-bf57-40aa-b395-1b40aedf3559\") " pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"
Jul 07 06:59:56.104414 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.104323    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f75f0412-bf57-40aa-b395-1b40aedf3559-home\") pod \"e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79\" (UID: \"f75f0412-bf57-40aa-b395-1b40aedf3559\") " pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"
Jul 07 06:59:56.104414 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.104339    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f75f0412-bf57-40aa-b395-1b40aedf3559-model-cache\") pod \"e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79\" (UID: \"f75f0412-bf57-40aa-b395-1b40aedf3559\") " pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"
Jul 07 06:59:56.104414 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.104358    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f75f0412-bf57-40aa-b395-1b40aedf3559-tls-certs\") pod \"e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79\" (UID: \"f75f0412-bf57-40aa-b395-1b40aedf3559\") " pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"
Jul 07 06:59:56.104530 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.104414    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2gc7\" (UniqueName: \"kubernetes.io/projected/f75f0412-bf57-40aa-b395-1b40aedf3559-kube-api-access-s2gc7\") pod \"e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79\" (UID: \"f75f0412-bf57-40aa-b395-1b40aedf3559\") " pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"
Jul 07 06:59:56.104530 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.104438    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f75f0412-bf57-40aa-b395-1b40aedf3559-tmp-dir\") pod \"e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79\" (UID: \"f75f0412-bf57-40aa-b395-1b40aedf3559\") " pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"
Jul 07 06:59:56.110679 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.110624    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"]
Jul 07 06:59:56.205131 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.205099    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f75f0412-bf57-40aa-b395-1b40aedf3559-dshm\") pod \"e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79\" (UID: \"f75f0412-bf57-40aa-b395-1b40aedf3559\") " pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"
Jul 07 06:59:56.205131 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.205133    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f75f0412-bf57-40aa-b395-1b40aedf3559-home\") pod \"e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79\" (UID: \"f75f0412-bf57-40aa-b395-1b40aedf3559\") " pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"
Jul 07 06:59:56.205394 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.205160    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f75f0412-bf57-40aa-b395-1b40aedf3559-model-cache\") pod \"e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79\" (UID: \"f75f0412-bf57-40aa-b395-1b40aedf3559\") " pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"
Jul 07 06:59:56.205394 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.205188    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f75f0412-bf57-40aa-b395-1b40aedf3559-tls-certs\") pod \"e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79\" (UID: \"f75f0412-bf57-40aa-b395-1b40aedf3559\") " pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"
Jul 07 06:59:56.205394 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.205276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2gc7\" (UniqueName: \"kubernetes.io/projected/f75f0412-bf57-40aa-b395-1b40aedf3559-kube-api-access-s2gc7\") pod \"e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79\" (UID: \"f75f0412-bf57-40aa-b395-1b40aedf3559\") " pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"
Jul 07 06:59:56.205394 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.205309    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f75f0412-bf57-40aa-b395-1b40aedf3559-tmp-dir\") pod \"e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79\" (UID: \"f75f0412-bf57-40aa-b395-1b40aedf3559\") " pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"
Jul 07 06:59:56.205605 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.205557    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f75f0412-bf57-40aa-b395-1b40aedf3559-home\") pod \"e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79\" (UID: \"f75f0412-bf57-40aa-b395-1b40aedf3559\") " pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"
Jul 07 06:59:56.205681 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.205665    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f75f0412-bf57-40aa-b395-1b40aedf3559-model-cache\") pod \"e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79\" (UID: \"f75f0412-bf57-40aa-b395-1b40aedf3559\") " pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"
Jul 07 06:59:56.205876 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.205840    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f75f0412-bf57-40aa-b395-1b40aedf3559-tmp-dir\") pod \"e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79\" (UID: \"f75f0412-bf57-40aa-b395-1b40aedf3559\") " pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"
Jul 07 06:59:56.207626 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.207604    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f75f0412-bf57-40aa-b395-1b40aedf3559-dshm\") pod \"e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79\" (UID: \"f75f0412-bf57-40aa-b395-1b40aedf3559\") " pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"
Jul 07 06:59:56.207885 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.207867    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f75f0412-bf57-40aa-b395-1b40aedf3559-tls-certs\") pod \"e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79\" (UID: \"f75f0412-bf57-40aa-b395-1b40aedf3559\") " pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"
Jul 07 06:59:56.223592 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.223561    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2gc7\" (UniqueName: \"kubernetes.io/projected/f75f0412-bf57-40aa-b395-1b40aedf3559-kube-api-access-s2gc7\") pod \"e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79\" (UID: \"f75f0412-bf57-40aa-b395-1b40aedf3559\") " pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"
Jul 07 06:59:56.406926 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.406846    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"
Jul 07 06:59:56.542821 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.542792    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"]
Jul 07 06:59:56.544976 ip-10-0-133-92 kubenswrapper[2563]: W0707 06:59:56.544950    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf75f0412_bf57_40aa_b395_1b40aedf3559.slice/crio-125943dc07765c469f2aeef36f63d8a03cc0659eaa0541432b770957ecad3dac WatchSource:0}: Error finding container 125943dc07765c469f2aeef36f63d8a03cc0659eaa0541432b770957ecad3dac: Status 404 returned error can't find the container with id 125943dc07765c469f2aeef36f63d8a03cc0659eaa0541432b770957ecad3dac
Jul 07 06:59:56.811990 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.811961    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79" event={"ID":"f75f0412-bf57-40aa-b395-1b40aedf3559","Type":"ContainerStarted","Data":"4b5e92c44303770f7de49db8d9c8834c4c84f59485a916c2540b14bea0be4b5a"}
Jul 07 06:59:56.812147 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.812002    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79" event={"ID":"f75f0412-bf57-40aa-b395-1b40aedf3559","Type":"ContainerStarted","Data":"125943dc07765c469f2aeef36f63d8a03cc0659eaa0541432b770957ecad3dac"}
Jul 07 06:59:56.830021 ip-10-0-133-92 kubenswrapper[2563]: I0707 06:59:56.829954    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79" podStartSLOduration=0.829928546 podStartE2EDuration="829.928546ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 06:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 06:59:56.829616924 +0000 UTC m=+1680.193813470" watchObservedRunningTime="2026-07-07 06:59:56.829928546 +0000 UTC m=+1680.194125091"
Jul 07 07:00:06.407161 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:06.407117    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"
Jul 07 07:00:06.407692 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:06.407179    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"
Jul 07 07:00:06.420297 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:06.420263    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"
Jul 07 07:00:06.869530 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:06.869502    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"
Jul 07 07:00:25.868517 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:25.868424    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"]
Jul 07 07:00:40.890227 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:40.890165    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79" podUID="f75f0412-bf57-40aa-b395-1b40aedf3559" containerName="main" containerID="cri-o://4b5e92c44303770f7de49db8d9c8834c4c84f59485a916c2540b14bea0be4b5a" gracePeriod=45
Jul 07 07:00:41.012772 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:41.012740    2563 generic.go:356] "Generic (PLEG): container finished" podID="f75f0412-bf57-40aa-b395-1b40aedf3559" containerID="4b5e92c44303770f7de49db8d9c8834c4c84f59485a916c2540b14bea0be4b5a" exitCode=0
Jul 07 07:00:41.012922 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:41.012783    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79" event={"ID":"f75f0412-bf57-40aa-b395-1b40aedf3559","Type":"ContainerDied","Data":"4b5e92c44303770f7de49db8d9c8834c4c84f59485a916c2540b14bea0be4b5a"}
Jul 07 07:00:41.134446 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:41.134421    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"
Jul 07 07:00:41.291308 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:41.291225    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f75f0412-bf57-40aa-b395-1b40aedf3559-home\") pod \"f75f0412-bf57-40aa-b395-1b40aedf3559\" (UID: \"f75f0412-bf57-40aa-b395-1b40aedf3559\") "
Jul 07 07:00:41.291308 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:41.291301    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f75f0412-bf57-40aa-b395-1b40aedf3559-tmp-dir\") pod \"f75f0412-bf57-40aa-b395-1b40aedf3559\" (UID: \"f75f0412-bf57-40aa-b395-1b40aedf3559\") "
Jul 07 07:00:41.291503 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:41.291340    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f75f0412-bf57-40aa-b395-1b40aedf3559-dshm\") pod \"f75f0412-bf57-40aa-b395-1b40aedf3559\" (UID: \"f75f0412-bf57-40aa-b395-1b40aedf3559\") "
Jul 07 07:00:41.291503 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:41.291366    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2gc7\" (UniqueName: \"kubernetes.io/projected/f75f0412-bf57-40aa-b395-1b40aedf3559-kube-api-access-s2gc7\") pod \"f75f0412-bf57-40aa-b395-1b40aedf3559\" (UID: \"f75f0412-bf57-40aa-b395-1b40aedf3559\") "
Jul 07 07:00:41.291503 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:41.291397    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f75f0412-bf57-40aa-b395-1b40aedf3559-model-cache\") pod \"f75f0412-bf57-40aa-b395-1b40aedf3559\" (UID: \"f75f0412-bf57-40aa-b395-1b40aedf3559\") "
Jul 07 07:00:41.291503 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:41.291455    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f75f0412-bf57-40aa-b395-1b40aedf3559-tls-certs\") pod \"f75f0412-bf57-40aa-b395-1b40aedf3559\" (UID: \"f75f0412-bf57-40aa-b395-1b40aedf3559\") "
Jul 07 07:00:41.291503 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:41.291456    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f75f0412-bf57-40aa-b395-1b40aedf3559-home" (OuterVolumeSpecName: "home") pod "f75f0412-bf57-40aa-b395-1b40aedf3559" (UID: "f75f0412-bf57-40aa-b395-1b40aedf3559"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:00:41.291767 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:41.291617    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f75f0412-bf57-40aa-b395-1b40aedf3559-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f75f0412-bf57-40aa-b395-1b40aedf3559" (UID: "f75f0412-bf57-40aa-b395-1b40aedf3559"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:00:41.291767 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:41.291716    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f75f0412-bf57-40aa-b395-1b40aedf3559-model-cache" (OuterVolumeSpecName: "model-cache") pod "f75f0412-bf57-40aa-b395-1b40aedf3559" (UID: "f75f0412-bf57-40aa-b395-1b40aedf3559"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:00:41.291879 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:41.291822    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f75f0412-bf57-40aa-b395-1b40aedf3559-model-cache\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:00:41.291879 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:41.291840    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f75f0412-bf57-40aa-b395-1b40aedf3559-home\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:00:41.291879 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:41.291874    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f75f0412-bf57-40aa-b395-1b40aedf3559-tmp-dir\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:00:41.293857 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:41.293836    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f75f0412-bf57-40aa-b395-1b40aedf3559-dshm" (OuterVolumeSpecName: "dshm") pod "f75f0412-bf57-40aa-b395-1b40aedf3559" (UID: "f75f0412-bf57-40aa-b395-1b40aedf3559"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:00:41.294052 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:41.294033    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f75f0412-bf57-40aa-b395-1b40aedf3559-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f75f0412-bf57-40aa-b395-1b40aedf3559" (UID: "f75f0412-bf57-40aa-b395-1b40aedf3559"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 07:00:41.294113 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:41.294050    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f75f0412-bf57-40aa-b395-1b40aedf3559-kube-api-access-s2gc7" (OuterVolumeSpecName: "kube-api-access-s2gc7") pod "f75f0412-bf57-40aa-b395-1b40aedf3559" (UID: "f75f0412-bf57-40aa-b395-1b40aedf3559"). InnerVolumeSpecName "kube-api-access-s2gc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 07:00:41.392476 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:41.392451    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f75f0412-bf57-40aa-b395-1b40aedf3559-tls-certs\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:00:41.392476 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:41.392472    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f75f0412-bf57-40aa-b395-1b40aedf3559-dshm\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:00:41.392601 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:41.392480    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2gc7\" (UniqueName: \"kubernetes.io/projected/f75f0412-bf57-40aa-b395-1b40aedf3559-kube-api-access-s2gc7\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:00:42.018914 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:42.018887    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"
Jul 07 07:00:42.019380 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:42.018925    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79" event={"ID":"f75f0412-bf57-40aa-b395-1b40aedf3559","Type":"ContainerDied","Data":"125943dc07765c469f2aeef36f63d8a03cc0659eaa0541432b770957ecad3dac"}
Jul 07 07:00:42.019380 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:42.018988    2563 scope.go:117] "RemoveContainer" containerID="4b5e92c44303770f7de49db8d9c8834c4c84f59485a916c2540b14bea0be4b5a"
Jul 07 07:00:42.041317 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:42.041290    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"]
Jul 07 07:00:42.048284 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:42.048262    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-f08145e7/e2e-lifecycle-model-f08145e7-kserve-7b8674d69f-22h79"]
Jul 07 07:00:43.171829 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:00:43.171801    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f75f0412-bf57-40aa-b395-1b40aedf3559" path="/var/lib/kubelet/pods/f75f0412-bf57-40aa-b395-1b40aedf3559/volumes"
Jul 07 07:02:43.029165 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.029119    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"]
Jul 07 07:02:43.029744 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.029570    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f75f0412-bf57-40aa-b395-1b40aedf3559" containerName="main"
Jul 07 07:02:43.029744 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.029591    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f75f0412-bf57-40aa-b395-1b40aedf3559" containerName="main"
Jul 07 07:02:43.029744 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.029682    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f75f0412-bf57-40aa-b395-1b40aedf3559" containerName="main"
Jul 07 07:02:43.032111 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.032087    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"
Jul 07 07:02:43.035677 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.035630    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9b5a3c\"/\"kube-root-ca.crt\""
Jul 07 07:02:43.035814 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.035630    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9b5a3c\"/\"test-model-9b5a3c-kserve-self-signed-certs\""
Jul 07 07:02:43.035814 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.035692    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9b5a3c\"/\"default-dockercfg-64djl\""
Jul 07 07:02:43.035814 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.035630    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9b5a3c\"/\"openshift-service-ca.crt\""
Jul 07 07:02:43.041621 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.041601    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"]
Jul 07 07:02:43.158109 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.158082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1d389f3-d645-44bf-a278-d68111eca928-model-cache\") pod \"test-model-9b5a3c-kserve-8ccb6b79f-649xl\" (UID: \"d1d389f3-d645-44bf-a278-d68111eca928\") " pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"
Jul 07 07:02:43.158109 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.158111    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1d389f3-d645-44bf-a278-d68111eca928-home\") pod \"test-model-9b5a3c-kserve-8ccb6b79f-649xl\" (UID: \"d1d389f3-d645-44bf-a278-d68111eca928\") " pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"
Jul 07 07:02:43.158272 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.158129    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1d389f3-d645-44bf-a278-d68111eca928-tls-certs\") pod \"test-model-9b5a3c-kserve-8ccb6b79f-649xl\" (UID: \"d1d389f3-d645-44bf-a278-d68111eca928\") " pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"
Jul 07 07:02:43.158272 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.158149    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1d389f3-d645-44bf-a278-d68111eca928-dshm\") pod \"test-model-9b5a3c-kserve-8ccb6b79f-649xl\" (UID: \"d1d389f3-d645-44bf-a278-d68111eca928\") " pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"
Jul 07 07:02:43.158379 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.158327    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1d389f3-d645-44bf-a278-d68111eca928-tmp-dir\") pod \"test-model-9b5a3c-kserve-8ccb6b79f-649xl\" (UID: \"d1d389f3-d645-44bf-a278-d68111eca928\") " pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"
Jul 07 07:02:43.158434 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.158387    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmvf6\" (UniqueName: \"kubernetes.io/projected/d1d389f3-d645-44bf-a278-d68111eca928-kube-api-access-dmvf6\") pod \"test-model-9b5a3c-kserve-8ccb6b79f-649xl\" (UID: \"d1d389f3-d645-44bf-a278-d68111eca928\") " pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"
Jul 07 07:02:43.259610 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.259581    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmvf6\" (UniqueName: \"kubernetes.io/projected/d1d389f3-d645-44bf-a278-d68111eca928-kube-api-access-dmvf6\") pod \"test-model-9b5a3c-kserve-8ccb6b79f-649xl\" (UID: \"d1d389f3-d645-44bf-a278-d68111eca928\") " pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"
Jul 07 07:02:43.259732 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.259619    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1d389f3-d645-44bf-a278-d68111eca928-model-cache\") pod \"test-model-9b5a3c-kserve-8ccb6b79f-649xl\" (UID: \"d1d389f3-d645-44bf-a278-d68111eca928\") " pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"
Jul 07 07:02:43.259732 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.259661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1d389f3-d645-44bf-a278-d68111eca928-home\") pod \"test-model-9b5a3c-kserve-8ccb6b79f-649xl\" (UID: \"d1d389f3-d645-44bf-a278-d68111eca928\") " pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"
Jul 07 07:02:43.259732 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.259686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1d389f3-d645-44bf-a278-d68111eca928-tls-certs\") pod \"test-model-9b5a3c-kserve-8ccb6b79f-649xl\" (UID: \"d1d389f3-d645-44bf-a278-d68111eca928\") " pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"
Jul 07 07:02:43.259732 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.259712    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1d389f3-d645-44bf-a278-d68111eca928-dshm\") pod \"test-model-9b5a3c-kserve-8ccb6b79f-649xl\" (UID: \"d1d389f3-d645-44bf-a278-d68111eca928\") " pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"
Jul 07 07:02:43.259931 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.259791    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1d389f3-d645-44bf-a278-d68111eca928-tmp-dir\") pod \"test-model-9b5a3c-kserve-8ccb6b79f-649xl\" (UID: \"d1d389f3-d645-44bf-a278-d68111eca928\") " pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"
Jul 07 07:02:43.260025 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.260004    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1d389f3-d645-44bf-a278-d68111eca928-model-cache\") pod \"test-model-9b5a3c-kserve-8ccb6b79f-649xl\" (UID: \"d1d389f3-d645-44bf-a278-d68111eca928\") " pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"
Jul 07 07:02:43.260025 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.260022    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1d389f3-d645-44bf-a278-d68111eca928-home\") pod \"test-model-9b5a3c-kserve-8ccb6b79f-649xl\" (UID: \"d1d389f3-d645-44bf-a278-d68111eca928\") " pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"
Jul 07 07:02:43.260130 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.260052    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1d389f3-d645-44bf-a278-d68111eca928-tmp-dir\") pod \"test-model-9b5a3c-kserve-8ccb6b79f-649xl\" (UID: \"d1d389f3-d645-44bf-a278-d68111eca928\") " pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"
Jul 07 07:02:43.262202 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.262177    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1d389f3-d645-44bf-a278-d68111eca928-dshm\") pod \"test-model-9b5a3c-kserve-8ccb6b79f-649xl\" (UID: \"d1d389f3-d645-44bf-a278-d68111eca928\") " pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"
Jul 07 07:02:43.262395 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.262375    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1d389f3-d645-44bf-a278-d68111eca928-tls-certs\") pod \"test-model-9b5a3c-kserve-8ccb6b79f-649xl\" (UID: \"d1d389f3-d645-44bf-a278-d68111eca928\") " pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"
Jul 07 07:02:43.268277 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.268253    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmvf6\" (UniqueName: \"kubernetes.io/projected/d1d389f3-d645-44bf-a278-d68111eca928-kube-api-access-dmvf6\") pod \"test-model-9b5a3c-kserve-8ccb6b79f-649xl\" (UID: \"d1d389f3-d645-44bf-a278-d68111eca928\") " pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"
Jul 07 07:02:43.341621 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.341600    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"
Jul 07 07:02:43.472481 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.472449    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"]
Jul 07 07:02:43.475292 ip-10-0-133-92 kubenswrapper[2563]: W0707 07:02:43.475265    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1d389f3_d645_44bf_a278_d68111eca928.slice/crio-0e0f2a682de6d26b9368e28795df3e6f2f580fdfabd143e0a33f9f2f40cc5428 WatchSource:0}: Error finding container 0e0f2a682de6d26b9368e28795df3e6f2f580fdfabd143e0a33f9f2f40cc5428: Status 404 returned error can't find the container with id 0e0f2a682de6d26b9368e28795df3e6f2f580fdfabd143e0a33f9f2f40cc5428
Jul 07 07:02:43.564582 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.564547    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl" event={"ID":"d1d389f3-d645-44bf-a278-d68111eca928","Type":"ContainerStarted","Data":"9b91d986997e69fbe88da081a039667113d7c89602a7a58b47743c26a4b2871d"}
Jul 07 07:02:43.564759 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.564593    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl" event={"ID":"d1d389f3-d645-44bf-a278-d68111eca928","Type":"ContainerStarted","Data":"0e0f2a682de6d26b9368e28795df3e6f2f580fdfabd143e0a33f9f2f40cc5428"}
Jul 07 07:02:43.586044 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:43.585946    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl" podStartSLOduration=0.585926107 podStartE2EDuration="585.926107ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:02:43.584220206 +0000 UTC m=+1846.948416748" watchObservedRunningTime="2026-07-07 07:02:43.585926107 +0000 UTC m=+1846.950122654"
Jul 07 07:02:53.342215 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:53.342173    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"
Jul 07 07:02:53.342215 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:53.342224    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"
Jul 07 07:02:53.355110 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:53.355077    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"
Jul 07 07:02:53.618302 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:02:53.618209    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"
Jul 07 07:03:14.020939 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.020906    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"]
Jul 07 07:03:14.024227 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.024208    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"
Jul 07 07:03:14.028914 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.028882    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-da1ed1\"/\"openshift-service-ca.crt\""
Jul 07 07:03:14.028914 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.028901    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-da1ed1\"/\"test-model-da1ed1-kserve-self-signed-certs\""
Jul 07 07:03:14.029100 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.028906    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-da1ed1\"/\"kube-root-ca.crt\""
Jul 07 07:03:14.029100 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.028974    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-da1ed1\"/\"default-dockercfg-9fz2t\""
Jul 07 07:03:14.037419 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.037391    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"]
Jul 07 07:03:14.138333 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.138295    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e88e7990-af54-40e4-b32a-8a948c106ba2-model-cache\") pod \"test-model-da1ed1-kserve-c4b84dff7-dcmfv\" (UID: \"e88e7990-af54-40e4-b32a-8a948c106ba2\") " pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"
Jul 07 07:03:14.138333 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.138339    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e88e7990-af54-40e4-b32a-8a948c106ba2-home\") pod \"test-model-da1ed1-kserve-c4b84dff7-dcmfv\" (UID: \"e88e7990-af54-40e4-b32a-8a948c106ba2\") " pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"
Jul 07 07:03:14.138542 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.138363    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e88e7990-af54-40e4-b32a-8a948c106ba2-tls-certs\") pod \"test-model-da1ed1-kserve-c4b84dff7-dcmfv\" (UID: \"e88e7990-af54-40e4-b32a-8a948c106ba2\") " pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"
Jul 07 07:03:14.138542 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.138390    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbsbz\" (UniqueName: \"kubernetes.io/projected/e88e7990-af54-40e4-b32a-8a948c106ba2-kube-api-access-bbsbz\") pod \"test-model-da1ed1-kserve-c4b84dff7-dcmfv\" (UID: \"e88e7990-af54-40e4-b32a-8a948c106ba2\") " pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"
Jul 07 07:03:14.138542 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.138457    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e88e7990-af54-40e4-b32a-8a948c106ba2-tmp-dir\") pod \"test-model-da1ed1-kserve-c4b84dff7-dcmfv\" (UID: \"e88e7990-af54-40e4-b32a-8a948c106ba2\") " pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"
Jul 07 07:03:14.138655 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.138563    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e88e7990-af54-40e4-b32a-8a948c106ba2-dshm\") pod \"test-model-da1ed1-kserve-c4b84dff7-dcmfv\" (UID: \"e88e7990-af54-40e4-b32a-8a948c106ba2\") " pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"
Jul 07 07:03:14.239653 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.239605    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e88e7990-af54-40e4-b32a-8a948c106ba2-home\") pod \"test-model-da1ed1-kserve-c4b84dff7-dcmfv\" (UID: \"e88e7990-af54-40e4-b32a-8a948c106ba2\") " pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"
Jul 07 07:03:14.239856 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.239697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e88e7990-af54-40e4-b32a-8a948c106ba2-tls-certs\") pod \"test-model-da1ed1-kserve-c4b84dff7-dcmfv\" (UID: \"e88e7990-af54-40e4-b32a-8a948c106ba2\") " pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"
Jul 07 07:03:14.239856 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.239726    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbsbz\" (UniqueName: \"kubernetes.io/projected/e88e7990-af54-40e4-b32a-8a948c106ba2-kube-api-access-bbsbz\") pod \"test-model-da1ed1-kserve-c4b84dff7-dcmfv\" (UID: \"e88e7990-af54-40e4-b32a-8a948c106ba2\") " pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"
Jul 07 07:03:14.239856 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.239766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e88e7990-af54-40e4-b32a-8a948c106ba2-tmp-dir\") pod \"test-model-da1ed1-kserve-c4b84dff7-dcmfv\" (UID: \"e88e7990-af54-40e4-b32a-8a948c106ba2\") " pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"
Jul 07 07:03:14.240016 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.239885    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e88e7990-af54-40e4-b32a-8a948c106ba2-dshm\") pod \"test-model-da1ed1-kserve-c4b84dff7-dcmfv\" (UID: \"e88e7990-af54-40e4-b32a-8a948c106ba2\") " pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"
Jul 07 07:03:14.240016 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.239915    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e88e7990-af54-40e4-b32a-8a948c106ba2-model-cache\") pod \"test-model-da1ed1-kserve-c4b84dff7-dcmfv\" (UID: \"e88e7990-af54-40e4-b32a-8a948c106ba2\") " pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"
Jul 07 07:03:14.240121 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.240046    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e88e7990-af54-40e4-b32a-8a948c106ba2-home\") pod \"test-model-da1ed1-kserve-c4b84dff7-dcmfv\" (UID: \"e88e7990-af54-40e4-b32a-8a948c106ba2\") " pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"
Jul 07 07:03:14.240178 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.240143    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e88e7990-af54-40e4-b32a-8a948c106ba2-tmp-dir\") pod \"test-model-da1ed1-kserve-c4b84dff7-dcmfv\" (UID: \"e88e7990-af54-40e4-b32a-8a948c106ba2\") " pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"
Jul 07 07:03:14.240272 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.240251    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e88e7990-af54-40e4-b32a-8a948c106ba2-model-cache\") pod \"test-model-da1ed1-kserve-c4b84dff7-dcmfv\" (UID: \"e88e7990-af54-40e4-b32a-8a948c106ba2\") " pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"
Jul 07 07:03:14.242293 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.242270    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e88e7990-af54-40e4-b32a-8a948c106ba2-dshm\") pod \"test-model-da1ed1-kserve-c4b84dff7-dcmfv\" (UID: \"e88e7990-af54-40e4-b32a-8a948c106ba2\") " pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"
Jul 07 07:03:14.242476 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.242460    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e88e7990-af54-40e4-b32a-8a948c106ba2-tls-certs\") pod \"test-model-da1ed1-kserve-c4b84dff7-dcmfv\" (UID: \"e88e7990-af54-40e4-b32a-8a948c106ba2\") " pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"
Jul 07 07:03:14.250022 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.249986    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbsbz\" (UniqueName: \"kubernetes.io/projected/e88e7990-af54-40e4-b32a-8a948c106ba2-kube-api-access-bbsbz\") pod \"test-model-da1ed1-kserve-c4b84dff7-dcmfv\" (UID: \"e88e7990-af54-40e4-b32a-8a948c106ba2\") " pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"
Jul 07 07:03:14.333699 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.333630    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"
Jul 07 07:03:14.491869 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.491842    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"]
Jul 07 07:03:14.494583 ip-10-0-133-92 kubenswrapper[2563]: W0707 07:03:14.494551    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode88e7990_af54_40e4_b32a_8a948c106ba2.slice/crio-01d49d7040d6c6a3749c0df0e31f3473ca367ac9a81bdcb7a19a4f678efad320 WatchSource:0}: Error finding container 01d49d7040d6c6a3749c0df0e31f3473ca367ac9a81bdcb7a19a4f678efad320: Status 404 returned error can't find the container with id 01d49d7040d6c6a3749c0df0e31f3473ca367ac9a81bdcb7a19a4f678efad320
Jul 07 07:03:14.706170 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.706083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv" event={"ID":"e88e7990-af54-40e4-b32a-8a948c106ba2","Type":"ContainerStarted","Data":"19ba80aae14978fa626d72818414308495a983cbd8bd6a0ffc2d75aec85d462c"}
Jul 07 07:03:14.706170 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.706121    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv" event={"ID":"e88e7990-af54-40e4-b32a-8a948c106ba2","Type":"ContainerStarted","Data":"01d49d7040d6c6a3749c0df0e31f3473ca367ac9a81bdcb7a19a4f678efad320"}
Jul 07 07:03:14.741127 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:14.741066    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv" podStartSLOduration=1.741047584 podStartE2EDuration="1.741047584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:03:14.738605925 +0000 UTC m=+1878.102802470" watchObservedRunningTime="2026-07-07 07:03:14.741047584 +0000 UTC m=+1878.105244128"
Jul 07 07:03:24.334813 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:24.334706    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"
Jul 07 07:03:24.334813 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:24.334795    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"
Jul 07 07:03:24.347643 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:24.347612    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"
Jul 07 07:03:24.763065 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:24.762980    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"
Jul 07 07:03:39.670603 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:39.670561    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"]
Jul 07 07:03:40.585958 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:40.585927    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"]
Jul 07 07:03:54.690890 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:54.690831    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl" podUID="d1d389f3-d645-44bf-a278-d68111eca928" containerName="main" containerID="cri-o://9b91d986997e69fbe88da081a039667113d7c89602a7a58b47743c26a4b2871d" gracePeriod=45
Jul 07 07:03:54.884999 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:54.884970    2563 generic.go:356] "Generic (PLEG): container finished" podID="d1d389f3-d645-44bf-a278-d68111eca928" containerID="9b91d986997e69fbe88da081a039667113d7c89602a7a58b47743c26a4b2871d" exitCode=0
Jul 07 07:03:54.885121 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:54.885047    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl" event={"ID":"d1d389f3-d645-44bf-a278-d68111eca928","Type":"ContainerDied","Data":"9b91d986997e69fbe88da081a039667113d7c89602a7a58b47743c26a4b2871d"}
Jul 07 07:03:54.942383 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:54.942327    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"
Jul 07 07:03:55.020729 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.020690    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1d389f3-d645-44bf-a278-d68111eca928-tmp-dir\") pod \"d1d389f3-d645-44bf-a278-d68111eca928\" (UID: \"d1d389f3-d645-44bf-a278-d68111eca928\") "
Jul 07 07:03:55.020729 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.020728    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1d389f3-d645-44bf-a278-d68111eca928-home\") pod \"d1d389f3-d645-44bf-a278-d68111eca928\" (UID: \"d1d389f3-d645-44bf-a278-d68111eca928\") "
Jul 07 07:03:55.020935 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.020778    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1d389f3-d645-44bf-a278-d68111eca928-tls-certs\") pod \"d1d389f3-d645-44bf-a278-d68111eca928\" (UID: \"d1d389f3-d645-44bf-a278-d68111eca928\") "
Jul 07 07:03:55.020935 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.020805    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1d389f3-d645-44bf-a278-d68111eca928-model-cache\") pod \"d1d389f3-d645-44bf-a278-d68111eca928\" (UID: \"d1d389f3-d645-44bf-a278-d68111eca928\") "
Jul 07 07:03:55.020935 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.020826    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmvf6\" (UniqueName: \"kubernetes.io/projected/d1d389f3-d645-44bf-a278-d68111eca928-kube-api-access-dmvf6\") pod \"d1d389f3-d645-44bf-a278-d68111eca928\" (UID: \"d1d389f3-d645-44bf-a278-d68111eca928\") "
Jul 07 07:03:55.020935 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.020906    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1d389f3-d645-44bf-a278-d68111eca928-dshm\") pod \"d1d389f3-d645-44bf-a278-d68111eca928\" (UID: \"d1d389f3-d645-44bf-a278-d68111eca928\") "
Jul 07 07:03:55.021129 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.021016    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1d389f3-d645-44bf-a278-d68111eca928-home" (OuterVolumeSpecName: "home") pod "d1d389f3-d645-44bf-a278-d68111eca928" (UID: "d1d389f3-d645-44bf-a278-d68111eca928"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:03:55.021129 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.021021    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1d389f3-d645-44bf-a278-d68111eca928-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d1d389f3-d645-44bf-a278-d68111eca928" (UID: "d1d389f3-d645-44bf-a278-d68111eca928"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:03:55.021239 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.021131    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1d389f3-d645-44bf-a278-d68111eca928-model-cache" (OuterVolumeSpecName: "model-cache") pod "d1d389f3-d645-44bf-a278-d68111eca928" (UID: "d1d389f3-d645-44bf-a278-d68111eca928"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:03:55.021239 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.021216    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1d389f3-d645-44bf-a278-d68111eca928-model-cache\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:03:55.021239 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.021235    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1d389f3-d645-44bf-a278-d68111eca928-tmp-dir\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:03:55.021344 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.021249    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1d389f3-d645-44bf-a278-d68111eca928-home\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:03:55.023255 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.023230    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1d389f3-d645-44bf-a278-d68111eca928-kube-api-access-dmvf6" (OuterVolumeSpecName: "kube-api-access-dmvf6") pod "d1d389f3-d645-44bf-a278-d68111eca928" (UID: "d1d389f3-d645-44bf-a278-d68111eca928"). InnerVolumeSpecName "kube-api-access-dmvf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 07:03:55.023255 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.023246    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1d389f3-d645-44bf-a278-d68111eca928-dshm" (OuterVolumeSpecName: "dshm") pod "d1d389f3-d645-44bf-a278-d68111eca928" (UID: "d1d389f3-d645-44bf-a278-d68111eca928"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:03:55.023414 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.023248    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1d389f3-d645-44bf-a278-d68111eca928-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d1d389f3-d645-44bf-a278-d68111eca928" (UID: "d1d389f3-d645-44bf-a278-d68111eca928"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 07:03:55.122043 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.121983    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1d389f3-d645-44bf-a278-d68111eca928-tls-certs\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:03:55.122043 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.122035    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmvf6\" (UniqueName: \"kubernetes.io/projected/d1d389f3-d645-44bf-a278-d68111eca928-kube-api-access-dmvf6\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:03:55.122043 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.122045    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1d389f3-d645-44bf-a278-d68111eca928-dshm\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:03:55.608093 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.608054    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv" podUID="e88e7990-af54-40e4-b32a-8a948c106ba2" containerName="main" containerID="cri-o://19ba80aae14978fa626d72818414308495a983cbd8bd6a0ffc2d75aec85d462c" gracePeriod=45
Jul 07 07:03:55.854059 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.854030    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"
Jul 07 07:03:55.891962 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.891880    2563 generic.go:356] "Generic (PLEG): container finished" podID="e88e7990-af54-40e4-b32a-8a948c106ba2" containerID="19ba80aae14978fa626d72818414308495a983cbd8bd6a0ffc2d75aec85d462c" exitCode=0
Jul 07 07:03:55.892111 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.891980    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"
Jul 07 07:03:55.892111 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.891994    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv" event={"ID":"e88e7990-af54-40e4-b32a-8a948c106ba2","Type":"ContainerDied","Data":"19ba80aae14978fa626d72818414308495a983cbd8bd6a0ffc2d75aec85d462c"}
Jul 07 07:03:55.892111 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.892036    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv" event={"ID":"e88e7990-af54-40e4-b32a-8a948c106ba2","Type":"ContainerDied","Data":"01d49d7040d6c6a3749c0df0e31f3473ca367ac9a81bdcb7a19a4f678efad320"}
Jul 07 07:03:55.892111 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.892059    2563 scope.go:117] "RemoveContainer" containerID="19ba80aae14978fa626d72818414308495a983cbd8bd6a0ffc2d75aec85d462c"
Jul 07 07:03:55.893369 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.893348    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl" event={"ID":"d1d389f3-d645-44bf-a278-d68111eca928","Type":"ContainerDied","Data":"0e0f2a682de6d26b9368e28795df3e6f2f580fdfabd143e0a33f9f2f40cc5428"}
Jul 07 07:03:55.893448 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.893370    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"
Jul 07 07:03:55.900995 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.900974    2563 scope.go:117] "RemoveContainer" containerID="19ba80aae14978fa626d72818414308495a983cbd8bd6a0ffc2d75aec85d462c"
Jul 07 07:03:55.901242 ip-10-0-133-92 kubenswrapper[2563]: E0707 07:03:55.901223    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19ba80aae14978fa626d72818414308495a983cbd8bd6a0ffc2d75aec85d462c\": container with ID starting with 19ba80aae14978fa626d72818414308495a983cbd8bd6a0ffc2d75aec85d462c not found: ID does not exist" containerID="19ba80aae14978fa626d72818414308495a983cbd8bd6a0ffc2d75aec85d462c"
Jul 07 07:03:55.901290 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.901250    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ba80aae14978fa626d72818414308495a983cbd8bd6a0ffc2d75aec85d462c"} err="failed to get container status \"19ba80aae14978fa626d72818414308495a983cbd8bd6a0ffc2d75aec85d462c\": rpc error: code = NotFound desc = could not find container \"19ba80aae14978fa626d72818414308495a983cbd8bd6a0ffc2d75aec85d462c\": container with ID starting with 19ba80aae14978fa626d72818414308495a983cbd8bd6a0ffc2d75aec85d462c not found: ID does not exist"
Jul 07 07:03:55.901290 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.901266    2563 scope.go:117] "RemoveContainer" containerID="9b91d986997e69fbe88da081a039667113d7c89602a7a58b47743c26a4b2871d"
Jul 07 07:03:55.908787 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.908765    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"]
Jul 07 07:03:55.914845 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.914823    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-9b5a3c/test-model-9b5a3c-kserve-8ccb6b79f-649xl"]
Jul 07 07:03:55.930191 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.930166    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e88e7990-af54-40e4-b32a-8a948c106ba2-tmp-dir\") pod \"e88e7990-af54-40e4-b32a-8a948c106ba2\" (UID: \"e88e7990-af54-40e4-b32a-8a948c106ba2\") "
Jul 07 07:03:55.930297 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.930217    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e88e7990-af54-40e4-b32a-8a948c106ba2-model-cache\") pod \"e88e7990-af54-40e4-b32a-8a948c106ba2\" (UID: \"e88e7990-af54-40e4-b32a-8a948c106ba2\") "
Jul 07 07:03:55.930297 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.930258    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e88e7990-af54-40e4-b32a-8a948c106ba2-tls-certs\") pod \"e88e7990-af54-40e4-b32a-8a948c106ba2\" (UID: \"e88e7990-af54-40e4-b32a-8a948c106ba2\") "
Jul 07 07:03:55.930297 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.930276    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbsbz\" (UniqueName: \"kubernetes.io/projected/e88e7990-af54-40e4-b32a-8a948c106ba2-kube-api-access-bbsbz\") pod \"e88e7990-af54-40e4-b32a-8a948c106ba2\" (UID: \"e88e7990-af54-40e4-b32a-8a948c106ba2\") "
Jul 07 07:03:55.930451 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.930341    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e88e7990-af54-40e4-b32a-8a948c106ba2-home\") pod \"e88e7990-af54-40e4-b32a-8a948c106ba2\" (UID: \"e88e7990-af54-40e4-b32a-8a948c106ba2\") "
Jul 07 07:03:55.930451 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.930358    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e88e7990-af54-40e4-b32a-8a948c106ba2-dshm\") pod \"e88e7990-af54-40e4-b32a-8a948c106ba2\" (UID: \"e88e7990-af54-40e4-b32a-8a948c106ba2\") "
Jul 07 07:03:55.930551 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.930483    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88e7990-af54-40e4-b32a-8a948c106ba2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e88e7990-af54-40e4-b32a-8a948c106ba2" (UID: "e88e7990-af54-40e4-b32a-8a948c106ba2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:03:55.930551 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.930537    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88e7990-af54-40e4-b32a-8a948c106ba2-model-cache" (OuterVolumeSpecName: "model-cache") pod "e88e7990-af54-40e4-b32a-8a948c106ba2" (UID: "e88e7990-af54-40e4-b32a-8a948c106ba2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:03:55.930664 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.930618    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88e7990-af54-40e4-b32a-8a948c106ba2-home" (OuterVolumeSpecName: "home") pod "e88e7990-af54-40e4-b32a-8a948c106ba2" (UID: "e88e7990-af54-40e4-b32a-8a948c106ba2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:03:55.932694 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.932666    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e88e7990-af54-40e4-b32a-8a948c106ba2-kube-api-access-bbsbz" (OuterVolumeSpecName: "kube-api-access-bbsbz") pod "e88e7990-af54-40e4-b32a-8a948c106ba2" (UID: "e88e7990-af54-40e4-b32a-8a948c106ba2"). InnerVolumeSpecName "kube-api-access-bbsbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 07:03:55.932794 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.932698    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88e7990-af54-40e4-b32a-8a948c106ba2-dshm" (OuterVolumeSpecName: "dshm") pod "e88e7990-af54-40e4-b32a-8a948c106ba2" (UID: "e88e7990-af54-40e4-b32a-8a948c106ba2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:03:55.932794 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:55.932698    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e88e7990-af54-40e4-b32a-8a948c106ba2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e88e7990-af54-40e4-b32a-8a948c106ba2" (UID: "e88e7990-af54-40e4-b32a-8a948c106ba2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 07:03:56.030947 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:56.030915    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e88e7990-af54-40e4-b32a-8a948c106ba2-home\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:03:56.030947 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:56.030942    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e88e7990-af54-40e4-b32a-8a948c106ba2-dshm\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:03:56.030947 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:56.030950    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e88e7990-af54-40e4-b32a-8a948c106ba2-tmp-dir\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:03:56.030947 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:56.030960    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e88e7990-af54-40e4-b32a-8a948c106ba2-model-cache\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:03:56.031256 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:56.030968    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e88e7990-af54-40e4-b32a-8a948c106ba2-tls-certs\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:03:56.031256 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:56.030975    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbsbz\" (UniqueName: \"kubernetes.io/projected/e88e7990-af54-40e4-b32a-8a948c106ba2-kube-api-access-bbsbz\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:03:56.211745 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:56.211716    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"]
Jul 07 07:03:56.215707 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:56.215676    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-da1ed1/test-model-da1ed1-kserve-c4b84dff7-dcmfv"]
Jul 07 07:03:57.172159 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:57.172114    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1d389f3-d645-44bf-a278-d68111eca928" path="/var/lib/kubelet/pods/d1d389f3-d645-44bf-a278-d68111eca928/volumes"
Jul 07 07:03:57.172704 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:03:57.172684    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e88e7990-af54-40e4-b32a-8a948c106ba2" path="/var/lib/kubelet/pods/e88e7990-af54-40e4-b32a-8a948c106ba2/volumes"
Jul 07 07:05:44.785085 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.785055    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"]
Jul 07 07:05:44.785554 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.785439    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1d389f3-d645-44bf-a278-d68111eca928" containerName="main"
Jul 07 07:05:44.785554 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.785452    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1d389f3-d645-44bf-a278-d68111eca928" containerName="main"
Jul 07 07:05:44.785554 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.785473    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e88e7990-af54-40e4-b32a-8a948c106ba2" containerName="main"
Jul 07 07:05:44.785554 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.785478    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e88e7990-af54-40e4-b32a-8a948c106ba2" containerName="main"
Jul 07 07:05:44.785554 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.785531    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e88e7990-af54-40e4-b32a-8a948c106ba2" containerName="main"
Jul 07 07:05:44.785554 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.785541    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1d389f3-d645-44bf-a278-d68111eca928" containerName="main"
Jul 07 07:05:44.788718 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.788698    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"
Jul 07 07:05:44.808509 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.808481    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-94448f\"/\"kube-root-ca.crt\""
Jul 07 07:05:44.808651 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.808562    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-94448f\"/\"default-dockercfg-hm5f9\""
Jul 07 07:05:44.808651 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.808563    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-94448f\"/\"openshift-service-ca.crt\""
Jul 07 07:05:44.808773 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.808647    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-94448f\"/\"auth-test-model-94448f-kserve-self-signed-certs\""
Jul 07 07:05:44.810491 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.810468    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"]
Jul 07 07:05:44.850911 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.850882    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/644e1f6e-89b8-4594-ba66-60408a9c5636-home\") pod \"auth-test-model-94448f-kserve-7545896987-kr5z2\" (UID: \"644e1f6e-89b8-4594-ba66-60408a9c5636\") " pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"
Jul 07 07:05:44.851048 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.850925    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/644e1f6e-89b8-4594-ba66-60408a9c5636-tmp-dir\") pod \"auth-test-model-94448f-kserve-7545896987-kr5z2\" (UID: \"644e1f6e-89b8-4594-ba66-60408a9c5636\") " pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"
Jul 07 07:05:44.851048 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.850954    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m8lh\" (UniqueName: \"kubernetes.io/projected/644e1f6e-89b8-4594-ba66-60408a9c5636-kube-api-access-2m8lh\") pod \"auth-test-model-94448f-kserve-7545896987-kr5z2\" (UID: \"644e1f6e-89b8-4594-ba66-60408a9c5636\") " pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"
Jul 07 07:05:44.851048 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.851004    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/644e1f6e-89b8-4594-ba66-60408a9c5636-dshm\") pod \"auth-test-model-94448f-kserve-7545896987-kr5z2\" (UID: \"644e1f6e-89b8-4594-ba66-60408a9c5636\") " pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"
Jul 07 07:05:44.851048 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.851024    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/644e1f6e-89b8-4594-ba66-60408a9c5636-tls-certs\") pod \"auth-test-model-94448f-kserve-7545896987-kr5z2\" (UID: \"644e1f6e-89b8-4594-ba66-60408a9c5636\") " pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"
Jul 07 07:05:44.851211 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.851094    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/644e1f6e-89b8-4594-ba66-60408a9c5636-model-cache\") pod \"auth-test-model-94448f-kserve-7545896987-kr5z2\" (UID: \"644e1f6e-89b8-4594-ba66-60408a9c5636\") " pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"
Jul 07 07:05:44.952449 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.952421    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/644e1f6e-89b8-4594-ba66-60408a9c5636-tmp-dir\") pod \"auth-test-model-94448f-kserve-7545896987-kr5z2\" (UID: \"644e1f6e-89b8-4594-ba66-60408a9c5636\") " pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"
Jul 07 07:05:44.952577 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.952456    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m8lh\" (UniqueName: \"kubernetes.io/projected/644e1f6e-89b8-4594-ba66-60408a9c5636-kube-api-access-2m8lh\") pod \"auth-test-model-94448f-kserve-7545896987-kr5z2\" (UID: \"644e1f6e-89b8-4594-ba66-60408a9c5636\") " pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"
Jul 07 07:05:44.952577 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.952485    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/644e1f6e-89b8-4594-ba66-60408a9c5636-dshm\") pod \"auth-test-model-94448f-kserve-7545896987-kr5z2\" (UID: \"644e1f6e-89b8-4594-ba66-60408a9c5636\") " pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"
Jul 07 07:05:44.952577 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.952505    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/644e1f6e-89b8-4594-ba66-60408a9c5636-tls-certs\") pod \"auth-test-model-94448f-kserve-7545896987-kr5z2\" (UID: \"644e1f6e-89b8-4594-ba66-60408a9c5636\") " pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"
Jul 07 07:05:44.952577 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.952532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/644e1f6e-89b8-4594-ba66-60408a9c5636-model-cache\") pod \"auth-test-model-94448f-kserve-7545896987-kr5z2\" (UID: \"644e1f6e-89b8-4594-ba66-60408a9c5636\") " pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"
Jul 07 07:05:44.952577 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.952566    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/644e1f6e-89b8-4594-ba66-60408a9c5636-home\") pod \"auth-test-model-94448f-kserve-7545896987-kr5z2\" (UID: \"644e1f6e-89b8-4594-ba66-60408a9c5636\") " pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"
Jul 07 07:05:44.952943 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.952919    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/644e1f6e-89b8-4594-ba66-60408a9c5636-tmp-dir\") pod \"auth-test-model-94448f-kserve-7545896987-kr5z2\" (UID: \"644e1f6e-89b8-4594-ba66-60408a9c5636\") " pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"
Jul 07 07:05:44.953024 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.952971    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/644e1f6e-89b8-4594-ba66-60408a9c5636-home\") pod \"auth-test-model-94448f-kserve-7545896987-kr5z2\" (UID: \"644e1f6e-89b8-4594-ba66-60408a9c5636\") " pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"
Jul 07 07:05:44.953070 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.953056    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/644e1f6e-89b8-4594-ba66-60408a9c5636-model-cache\") pod \"auth-test-model-94448f-kserve-7545896987-kr5z2\" (UID: \"644e1f6e-89b8-4594-ba66-60408a9c5636\") " pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"
Jul 07 07:05:44.955449 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.955424    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/644e1f6e-89b8-4594-ba66-60408a9c5636-dshm\") pod \"auth-test-model-94448f-kserve-7545896987-kr5z2\" (UID: \"644e1f6e-89b8-4594-ba66-60408a9c5636\") " pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"
Jul 07 07:05:44.955672 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.955625    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/644e1f6e-89b8-4594-ba66-60408a9c5636-tls-certs\") pod \"auth-test-model-94448f-kserve-7545896987-kr5z2\" (UID: \"644e1f6e-89b8-4594-ba66-60408a9c5636\") " pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"
Jul 07 07:05:44.963173 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:44.963152    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m8lh\" (UniqueName: \"kubernetes.io/projected/644e1f6e-89b8-4594-ba66-60408a9c5636-kube-api-access-2m8lh\") pod \"auth-test-model-94448f-kserve-7545896987-kr5z2\" (UID: \"644e1f6e-89b8-4594-ba66-60408a9c5636\") " pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"
Jul 07 07:05:45.098453 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:45.098426    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"
Jul 07 07:05:45.238892 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:45.238868    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"]
Jul 07 07:05:45.241042 ip-10-0-133-92 kubenswrapper[2563]: W0707 07:05:45.241016    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod644e1f6e_89b8_4594_ba66_60408a9c5636.slice/crio-eb05852db65d0640a319b4c1748f51047c9272f6e15dec1ba8e4c3e013fae5e4 WatchSource:0}: Error finding container eb05852db65d0640a319b4c1748f51047c9272f6e15dec1ba8e4c3e013fae5e4: Status 404 returned error can't find the container with id eb05852db65d0640a319b4c1748f51047c9272f6e15dec1ba8e4c3e013fae5e4
Jul 07 07:05:45.390733 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:45.390644    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2" event={"ID":"644e1f6e-89b8-4594-ba66-60408a9c5636","Type":"ContainerStarted","Data":"9f5a852839a7021f90e66188bd7f08d6b27cfd30241c21ccc50e0c3adedea17d"}
Jul 07 07:05:45.390733 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:45.390686    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2" event={"ID":"644e1f6e-89b8-4594-ba66-60408a9c5636","Type":"ContainerStarted","Data":"eb05852db65d0640a319b4c1748f51047c9272f6e15dec1ba8e4c3e013fae5e4"}
Jul 07 07:05:45.418296 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:45.418250    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2" podStartSLOduration=1.418234958 podStartE2EDuration="1.418234958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:05:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:05:45.418102459 +0000 UTC m=+2028.782299005" watchObservedRunningTime="2026-07-07 07:05:45.418234958 +0000 UTC m=+2028.782431502"
Jul 07 07:05:55.099336 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:55.099301    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"
Jul 07 07:05:55.099754 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:55.099350    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"
Jul 07 07:05:55.111924 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:55.111898    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"
Jul 07 07:05:55.445504 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:05:55.445435    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"
Jul 07 07:06:30.509770 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:30.509696    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"]
Jul 07 07:06:30.512996 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:30.512977    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"
Jul 07 07:06:30.519978 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:30.519953    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c72c9a\"/\"kube-root-ca.crt\""
Jul 07 07:06:30.520081 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:30.519964    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c72c9a\"/\"default-dockercfg-m5thp\""
Jul 07 07:06:30.520151 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:30.519963    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c72c9a\"/\"auth-test-model-c72c9a-kserve-self-signed-certs\""
Jul 07 07:06:30.520151 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:30.520041    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c72c9a\"/\"openshift-service-ca.crt\""
Jul 07 07:06:30.530069 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:30.530048    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"]
Jul 07 07:06:30.631157 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:30.631126    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd2bv\" (UniqueName: \"kubernetes.io/projected/81878266-fc53-4fcd-ac35-80d62b8e7f5c-kube-api-access-gd2bv\") pod \"auth-test-model-c72c9a-kserve-694c7f9699-dcr5l\" (UID: \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\") " pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"
Jul 07 07:06:30.631303 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:30.631201    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81878266-fc53-4fcd-ac35-80d62b8e7f5c-dshm\") pod \"auth-test-model-c72c9a-kserve-694c7f9699-dcr5l\" (UID: \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\") " pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"
Jul 07 07:06:30.631303 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:30.631276    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81878266-fc53-4fcd-ac35-80d62b8e7f5c-model-cache\") pod \"auth-test-model-c72c9a-kserve-694c7f9699-dcr5l\" (UID: \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\") " pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"
Jul 07 07:06:30.631473 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:30.631333    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81878266-fc53-4fcd-ac35-80d62b8e7f5c-tls-certs\") pod \"auth-test-model-c72c9a-kserve-694c7f9699-dcr5l\" (UID: \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\") " pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"
Jul 07 07:06:30.631473 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:30.631399    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81878266-fc53-4fcd-ac35-80d62b8e7f5c-home\") pod \"auth-test-model-c72c9a-kserve-694c7f9699-dcr5l\" (UID: \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\") " pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"
Jul 07 07:06:30.631473 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:30.631426    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81878266-fc53-4fcd-ac35-80d62b8e7f5c-tmp-dir\") pod \"auth-test-model-c72c9a-kserve-694c7f9699-dcr5l\" (UID: \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\") " pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"
Jul 07 07:06:30.732485 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:30.732462    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd2bv\" (UniqueName: \"kubernetes.io/projected/81878266-fc53-4fcd-ac35-80d62b8e7f5c-kube-api-access-gd2bv\") pod \"auth-test-model-c72c9a-kserve-694c7f9699-dcr5l\" (UID: \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\") " pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"
Jul 07 07:06:30.732608 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:30.732509    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81878266-fc53-4fcd-ac35-80d62b8e7f5c-dshm\") pod \"auth-test-model-c72c9a-kserve-694c7f9699-dcr5l\" (UID: \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\") " pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"
Jul 07 07:06:30.732608 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:30.732577    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81878266-fc53-4fcd-ac35-80d62b8e7f5c-model-cache\") pod \"auth-test-model-c72c9a-kserve-694c7f9699-dcr5l\" (UID: \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\") " pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"
Jul 07 07:06:30.732742 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:30.732610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81878266-fc53-4fcd-ac35-80d62b8e7f5c-tls-certs\") pod \"auth-test-model-c72c9a-kserve-694c7f9699-dcr5l\" (UID: \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\") " pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"
Jul 07 07:06:30.732742 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:30.732654    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81878266-fc53-4fcd-ac35-80d62b8e7f5c-home\") pod \"auth-test-model-c72c9a-kserve-694c7f9699-dcr5l\" (UID: \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\") " pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"
Jul 07 07:06:30.732742 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:30.732680    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81878266-fc53-4fcd-ac35-80d62b8e7f5c-tmp-dir\") pod \"auth-test-model-c72c9a-kserve-694c7f9699-dcr5l\" (UID: \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\") " pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"
Jul 07 07:06:30.732998 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:30.732977    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81878266-fc53-4fcd-ac35-80d62b8e7f5c-model-cache\") pod \"auth-test-model-c72c9a-kserve-694c7f9699-dcr5l\" (UID: \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\") " pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"
Jul 07 07:06:30.733069 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:30.733022    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81878266-fc53-4fcd-ac35-80d62b8e7f5c-tmp-dir\") pod \"auth-test-model-c72c9a-kserve-694c7f9699-dcr5l\" (UID: \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\") " pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"
Jul 07 07:06:30.733170 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:30.733088    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81878266-fc53-4fcd-ac35-80d62b8e7f5c-home\") pod \"auth-test-model-c72c9a-kserve-694c7f9699-dcr5l\" (UID: \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\") " pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"
Jul 07 07:06:30.735033 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:30.735014    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81878266-fc53-4fcd-ac35-80d62b8e7f5c-dshm\") pod \"auth-test-model-c72c9a-kserve-694c7f9699-dcr5l\" (UID: \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\") " pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"
Jul 07 07:06:30.735337 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:30.735316    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81878266-fc53-4fcd-ac35-80d62b8e7f5c-tls-certs\") pod \"auth-test-model-c72c9a-kserve-694c7f9699-dcr5l\" (UID: \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\") " pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"
Jul 07 07:06:30.742871 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:30.742846    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd2bv\" (UniqueName: \"kubernetes.io/projected/81878266-fc53-4fcd-ac35-80d62b8e7f5c-kube-api-access-gd2bv\") pod \"auth-test-model-c72c9a-kserve-694c7f9699-dcr5l\" (UID: \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\") " pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"
Jul 07 07:06:30.822468 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:30.822443    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"
Jul 07 07:06:30.961450 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:30.961421    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"]
Jul 07 07:06:30.963347 ip-10-0-133-92 kubenswrapper[2563]: W0707 07:06:30.963322    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81878266_fc53_4fcd_ac35_80d62b8e7f5c.slice/crio-40d0d9ff059ee805a6d4d364b168d7e9da463b58daba19e7071152383fe2f77d WatchSource:0}: Error finding container 40d0d9ff059ee805a6d4d364b168d7e9da463b58daba19e7071152383fe2f77d: Status 404 returned error can't find the container with id 40d0d9ff059ee805a6d4d364b168d7e9da463b58daba19e7071152383fe2f77d
Jul 07 07:06:31.590303 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:31.590270    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l" event={"ID":"81878266-fc53-4fcd-ac35-80d62b8e7f5c","Type":"ContainerStarted","Data":"d6ca3523f89892f972f60b6ffec4d33cb876ac66a766d1404c60cec718601444"}
Jul 07 07:06:31.590677 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:31.590309    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l" event={"ID":"81878266-fc53-4fcd-ac35-80d62b8e7f5c","Type":"ContainerStarted","Data":"40d0d9ff059ee805a6d4d364b168d7e9da463b58daba19e7071152383fe2f77d"}
Jul 07 07:06:31.640274 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:31.640230    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l" podStartSLOduration=1.640215215 podStartE2EDuration="1.640215215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:06:31.638842609 +0000 UTC m=+2075.003039154" watchObservedRunningTime="2026-07-07 07:06:31.640215215 +0000 UTC m=+2075.004411797"
Jul 07 07:06:40.823257 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:40.823223    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"
Jul 07 07:06:40.823742 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:40.823269    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"
Jul 07 07:06:40.840096 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:40.840073    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"
Jul 07 07:06:41.651473 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:06:41.651447    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"
Jul 07 07:08:32.794243 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:32.794211    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"]
Jul 07 07:08:33.615411 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:33.615380    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"]
Jul 07 07:08:47.813963 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:47.813924    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2" podUID="644e1f6e-89b8-4594-ba66-60408a9c5636" containerName="main" containerID="cri-o://9f5a852839a7021f90e66188bd7f08d6b27cfd30241c21ccc50e0c3adedea17d" gracePeriod=45
Jul 07 07:08:48.053869 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.053844    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"
Jul 07 07:08:48.188337 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.188254    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/644e1f6e-89b8-4594-ba66-60408a9c5636-home\") pod \"644e1f6e-89b8-4594-ba66-60408a9c5636\" (UID: \"644e1f6e-89b8-4594-ba66-60408a9c5636\") "
Jul 07 07:08:48.188337 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.188296    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/644e1f6e-89b8-4594-ba66-60408a9c5636-model-cache\") pod \"644e1f6e-89b8-4594-ba66-60408a9c5636\" (UID: \"644e1f6e-89b8-4594-ba66-60408a9c5636\") "
Jul 07 07:08:48.188553 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.188363    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/644e1f6e-89b8-4594-ba66-60408a9c5636-dshm\") pod \"644e1f6e-89b8-4594-ba66-60408a9c5636\" (UID: \"644e1f6e-89b8-4594-ba66-60408a9c5636\") "
Jul 07 07:08:48.188553 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.188387    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2m8lh\" (UniqueName: \"kubernetes.io/projected/644e1f6e-89b8-4594-ba66-60408a9c5636-kube-api-access-2m8lh\") pod \"644e1f6e-89b8-4594-ba66-60408a9c5636\" (UID: \"644e1f6e-89b8-4594-ba66-60408a9c5636\") "
Jul 07 07:08:48.188553 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.188421    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/644e1f6e-89b8-4594-ba66-60408a9c5636-tmp-dir\") pod \"644e1f6e-89b8-4594-ba66-60408a9c5636\" (UID: \"644e1f6e-89b8-4594-ba66-60408a9c5636\") "
Jul 07 07:08:48.188553 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.188444    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/644e1f6e-89b8-4594-ba66-60408a9c5636-tls-certs\") pod \"644e1f6e-89b8-4594-ba66-60408a9c5636\" (UID: \"644e1f6e-89b8-4594-ba66-60408a9c5636\") "
Jul 07 07:08:48.188553 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.188502    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/644e1f6e-89b8-4594-ba66-60408a9c5636-home" (OuterVolumeSpecName: "home") pod "644e1f6e-89b8-4594-ba66-60408a9c5636" (UID: "644e1f6e-89b8-4594-ba66-60408a9c5636"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:08:48.188814 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.188676    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/644e1f6e-89b8-4594-ba66-60408a9c5636-model-cache" (OuterVolumeSpecName: "model-cache") pod "644e1f6e-89b8-4594-ba66-60408a9c5636" (UID: "644e1f6e-89b8-4594-ba66-60408a9c5636"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:08:48.188814 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.188722    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/644e1f6e-89b8-4594-ba66-60408a9c5636-home\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:08:48.188814 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.188739    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/644e1f6e-89b8-4594-ba66-60408a9c5636-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "644e1f6e-89b8-4594-ba66-60408a9c5636" (UID: "644e1f6e-89b8-4594-ba66-60408a9c5636"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:08:48.190815 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.190793    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/644e1f6e-89b8-4594-ba66-60408a9c5636-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "644e1f6e-89b8-4594-ba66-60408a9c5636" (UID: "644e1f6e-89b8-4594-ba66-60408a9c5636"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 07:08:48.190899 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.190839    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/644e1f6e-89b8-4594-ba66-60408a9c5636-kube-api-access-2m8lh" (OuterVolumeSpecName: "kube-api-access-2m8lh") pod "644e1f6e-89b8-4594-ba66-60408a9c5636" (UID: "644e1f6e-89b8-4594-ba66-60408a9c5636"). InnerVolumeSpecName "kube-api-access-2m8lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 07:08:48.190899 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.190864    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/644e1f6e-89b8-4594-ba66-60408a9c5636-dshm" (OuterVolumeSpecName: "dshm") pod "644e1f6e-89b8-4594-ba66-60408a9c5636" (UID: "644e1f6e-89b8-4594-ba66-60408a9c5636"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:08:48.217470 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.217444    2563 generic.go:356] "Generic (PLEG): container finished" podID="644e1f6e-89b8-4594-ba66-60408a9c5636" containerID="9f5a852839a7021f90e66188bd7f08d6b27cfd30241c21ccc50e0c3adedea17d" exitCode=0
Jul 07 07:08:48.217574 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.217493    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"
Jul 07 07:08:48.217574 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.217533    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2" event={"ID":"644e1f6e-89b8-4594-ba66-60408a9c5636","Type":"ContainerDied","Data":"9f5a852839a7021f90e66188bd7f08d6b27cfd30241c21ccc50e0c3adedea17d"}
Jul 07 07:08:48.217709 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.217576    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2" event={"ID":"644e1f6e-89b8-4594-ba66-60408a9c5636","Type":"ContainerDied","Data":"eb05852db65d0640a319b4c1748f51047c9272f6e15dec1ba8e4c3e013fae5e4"}
Jul 07 07:08:48.217709 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.217594    2563 scope.go:117] "RemoveContainer" containerID="9f5a852839a7021f90e66188bd7f08d6b27cfd30241c21ccc50e0c3adedea17d"
Jul 07 07:08:48.232895 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.232870    2563 scope.go:117] "RemoveContainer" containerID="9f5a852839a7021f90e66188bd7f08d6b27cfd30241c21ccc50e0c3adedea17d"
Jul 07 07:08:48.233158 ip-10-0-133-92 kubenswrapper[2563]: E0707 07:08:48.233135    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f5a852839a7021f90e66188bd7f08d6b27cfd30241c21ccc50e0c3adedea17d\": container with ID starting with 9f5a852839a7021f90e66188bd7f08d6b27cfd30241c21ccc50e0c3adedea17d not found: ID does not exist" containerID="9f5a852839a7021f90e66188bd7f08d6b27cfd30241c21ccc50e0c3adedea17d"
Jul 07 07:08:48.233219 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.233170    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5a852839a7021f90e66188bd7f08d6b27cfd30241c21ccc50e0c3adedea17d"} err="failed to get container status \"9f5a852839a7021f90e66188bd7f08d6b27cfd30241c21ccc50e0c3adedea17d\": rpc error: code = NotFound desc = could not find container \"9f5a852839a7021f90e66188bd7f08d6b27cfd30241c21ccc50e0c3adedea17d\": container with ID starting with 9f5a852839a7021f90e66188bd7f08d6b27cfd30241c21ccc50e0c3adedea17d not found: ID does not exist"
Jul 07 07:08:48.242682 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.242661    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"]
Jul 07 07:08:48.247305 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.247283    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-94448f/auth-test-model-94448f-kserve-7545896987-kr5z2"]
Jul 07 07:08:48.289560 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.289542    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/644e1f6e-89b8-4594-ba66-60408a9c5636-tmp-dir\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:08:48.289670 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.289562    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/644e1f6e-89b8-4594-ba66-60408a9c5636-tls-certs\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:08:48.289670 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.289574    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/644e1f6e-89b8-4594-ba66-60408a9c5636-model-cache\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:08:48.289670 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.289585    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/644e1f6e-89b8-4594-ba66-60408a9c5636-dshm\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:08:48.289670 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.289597    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m8lh\" (UniqueName: \"kubernetes.io/projected/644e1f6e-89b8-4594-ba66-60408a9c5636-kube-api-access-2m8lh\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:08:48.635519 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.635488    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l" podUID="81878266-fc53-4fcd-ac35-80d62b8e7f5c" containerName="main" containerID="cri-o://d6ca3523f89892f972f60b6ffec4d33cb876ac66a766d1404c60cec718601444" gracePeriod=45
Jul 07 07:08:48.886230 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.886176    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"
Jul 07 07:08:48.995051 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.995026    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81878266-fc53-4fcd-ac35-80d62b8e7f5c-tmp-dir\") pod \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\" (UID: \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\") "
Jul 07 07:08:48.995183 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.995064    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81878266-fc53-4fcd-ac35-80d62b8e7f5c-home\") pod \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\" (UID: \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\") "
Jul 07 07:08:48.995183 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.995084    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81878266-fc53-4fcd-ac35-80d62b8e7f5c-tls-certs\") pod \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\" (UID: \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\") "
Jul 07 07:08:48.995183 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.995117    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81878266-fc53-4fcd-ac35-80d62b8e7f5c-model-cache\") pod \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\" (UID: \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\") "
Jul 07 07:08:48.995183 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.995148    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gd2bv\" (UniqueName: \"kubernetes.io/projected/81878266-fc53-4fcd-ac35-80d62b8e7f5c-kube-api-access-gd2bv\") pod \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\" (UID: \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\") "
Jul 07 07:08:48.995365 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.995217    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81878266-fc53-4fcd-ac35-80d62b8e7f5c-dshm\") pod \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\" (UID: \"81878266-fc53-4fcd-ac35-80d62b8e7f5c\") "
Jul 07 07:08:48.995365 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.995317    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81878266-fc53-4fcd-ac35-80d62b8e7f5c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "81878266-fc53-4fcd-ac35-80d62b8e7f5c" (UID: "81878266-fc53-4fcd-ac35-80d62b8e7f5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:08:48.995365 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.995325    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81878266-fc53-4fcd-ac35-80d62b8e7f5c-home" (OuterVolumeSpecName: "home") pod "81878266-fc53-4fcd-ac35-80d62b8e7f5c" (UID: "81878266-fc53-4fcd-ac35-80d62b8e7f5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:08:48.995528 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.995388    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81878266-fc53-4fcd-ac35-80d62b8e7f5c-model-cache" (OuterVolumeSpecName: "model-cache") pod "81878266-fc53-4fcd-ac35-80d62b8e7f5c" (UID: "81878266-fc53-4fcd-ac35-80d62b8e7f5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:08:48.995528 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.995497    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81878266-fc53-4fcd-ac35-80d62b8e7f5c-home\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:08:48.995528 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.995509    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81878266-fc53-4fcd-ac35-80d62b8e7f5c-model-cache\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:08:48.995528 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.995519    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81878266-fc53-4fcd-ac35-80d62b8e7f5c-tmp-dir\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:08:48.997400 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.997364    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81878266-fc53-4fcd-ac35-80d62b8e7f5c-dshm" (OuterVolumeSpecName: "dshm") pod "81878266-fc53-4fcd-ac35-80d62b8e7f5c" (UID: "81878266-fc53-4fcd-ac35-80d62b8e7f5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:08:48.997492 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.997458    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81878266-fc53-4fcd-ac35-80d62b8e7f5c-kube-api-access-gd2bv" (OuterVolumeSpecName: "kube-api-access-gd2bv") pod "81878266-fc53-4fcd-ac35-80d62b8e7f5c" (UID: "81878266-fc53-4fcd-ac35-80d62b8e7f5c"). InnerVolumeSpecName "kube-api-access-gd2bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 07:08:48.997492 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:48.997476    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81878266-fc53-4fcd-ac35-80d62b8e7f5c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "81878266-fc53-4fcd-ac35-80d62b8e7f5c" (UID: "81878266-fc53-4fcd-ac35-80d62b8e7f5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 07:08:49.096549 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:49.096520    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81878266-fc53-4fcd-ac35-80d62b8e7f5c-tls-certs\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:08:49.096549 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:49.096547    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd2bv\" (UniqueName: \"kubernetes.io/projected/81878266-fc53-4fcd-ac35-80d62b8e7f5c-kube-api-access-gd2bv\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:08:49.096714 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:49.096561    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81878266-fc53-4fcd-ac35-80d62b8e7f5c-dshm\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:08:49.171974 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:49.171919    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="644e1f6e-89b8-4594-ba66-60408a9c5636" path="/var/lib/kubelet/pods/644e1f6e-89b8-4594-ba66-60408a9c5636/volumes"
Jul 07 07:08:49.223708 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:49.223684    2563 generic.go:356] "Generic (PLEG): container finished" podID="81878266-fc53-4fcd-ac35-80d62b8e7f5c" containerID="d6ca3523f89892f972f60b6ffec4d33cb876ac66a766d1404c60cec718601444" exitCode=0
Jul 07 07:08:49.223813 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:49.223716    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l" event={"ID":"81878266-fc53-4fcd-ac35-80d62b8e7f5c","Type":"ContainerDied","Data":"d6ca3523f89892f972f60b6ffec4d33cb876ac66a766d1404c60cec718601444"}
Jul 07 07:08:49.223813 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:49.223734    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"
Jul 07 07:08:49.223813 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:49.223750    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l" event={"ID":"81878266-fc53-4fcd-ac35-80d62b8e7f5c","Type":"ContainerDied","Data":"40d0d9ff059ee805a6d4d364b168d7e9da463b58daba19e7071152383fe2f77d"}
Jul 07 07:08:49.223813 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:49.223766    2563 scope.go:117] "RemoveContainer" containerID="d6ca3523f89892f972f60b6ffec4d33cb876ac66a766d1404c60cec718601444"
Jul 07 07:08:49.234351 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:49.234332    2563 scope.go:117] "RemoveContainer" containerID="d6ca3523f89892f972f60b6ffec4d33cb876ac66a766d1404c60cec718601444"
Jul 07 07:08:49.234596 ip-10-0-133-92 kubenswrapper[2563]: E0707 07:08:49.234576    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6ca3523f89892f972f60b6ffec4d33cb876ac66a766d1404c60cec718601444\": container with ID starting with d6ca3523f89892f972f60b6ffec4d33cb876ac66a766d1404c60cec718601444 not found: ID does not exist" containerID="d6ca3523f89892f972f60b6ffec4d33cb876ac66a766d1404c60cec718601444"
Jul 07 07:08:49.234681 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:49.234608    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6ca3523f89892f972f60b6ffec4d33cb876ac66a766d1404c60cec718601444"} err="failed to get container status \"d6ca3523f89892f972f60b6ffec4d33cb876ac66a766d1404c60cec718601444\": rpc error: code = NotFound desc = could not find container \"d6ca3523f89892f972f60b6ffec4d33cb876ac66a766d1404c60cec718601444\": container with ID starting with d6ca3523f89892f972f60b6ffec4d33cb876ac66a766d1404c60cec718601444 not found: ID does not exist"
Jul 07 07:08:49.241624 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:49.241603    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"]
Jul 07 07:08:49.245735 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:49.245709    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c72c9a/auth-test-model-c72c9a-kserve-694c7f9699-dcr5l"]
Jul 07 07:08:51.171728 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:51.171689    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81878266-fc53-4fcd-ac35-80d62b8e7f5c" path="/var/lib/kubelet/pods/81878266-fc53-4fcd-ac35-80d62b8e7f5c/volumes"
Jul 07 07:08:56.130029 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.129985    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"]
Jul 07 07:08:56.130479 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.130457    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="644e1f6e-89b8-4594-ba66-60408a9c5636" containerName="main"
Jul 07 07:08:56.130479 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.130475    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="644e1f6e-89b8-4594-ba66-60408a9c5636" containerName="main"
Jul 07 07:08:56.130555 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.130518    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81878266-fc53-4fcd-ac35-80d62b8e7f5c" containerName="main"
Jul 07 07:08:56.130555 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.130528    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="81878266-fc53-4fcd-ac35-80d62b8e7f5c" containerName="main"
Jul 07 07:08:56.130630 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.130616    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="644e1f6e-89b8-4594-ba66-60408a9c5636" containerName="main"
Jul 07 07:08:56.130682 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.130653    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="81878266-fc53-4fcd-ac35-80d62b8e7f5c" containerName="main"
Jul 07 07:08:56.135628 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.135612    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"
Jul 07 07:08:56.138954 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.138930    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-94448f\"/\"kube-root-ca.crt\""
Jul 07 07:08:56.139073 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.139057    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-94448f\"/\"default-dockercfg-hm5f9\""
Jul 07 07:08:56.139144 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.139073    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-94448f\"/\"openshift-service-ca.crt\""
Jul 07 07:08:56.139144 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.139109    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-94448f\"/\"sub-test-model-94448f-kserve-self-signed-certs\""
Jul 07 07:08:56.150929 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.150909    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"]
Jul 07 07:08:56.157299 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.157280    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27e508f3-1d9d-46ba-9683-61bd584e13b9-tmp-dir\") pod \"sub-test-model-94448f-kserve-865dd6f457-4982z\" (UID: \"27e508f3-1d9d-46ba-9683-61bd584e13b9\") " pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"
Jul 07 07:08:56.157398 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.157310    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27e508f3-1d9d-46ba-9683-61bd584e13b9-tls-certs\") pod \"sub-test-model-94448f-kserve-865dd6f457-4982z\" (UID: \"27e508f3-1d9d-46ba-9683-61bd584e13b9\") " pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"
Jul 07 07:08:56.157398 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.157343    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27e508f3-1d9d-46ba-9683-61bd584e13b9-home\") pod \"sub-test-model-94448f-kserve-865dd6f457-4982z\" (UID: \"27e508f3-1d9d-46ba-9683-61bd584e13b9\") " pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"
Jul 07 07:08:56.157398 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.157385    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27e508f3-1d9d-46ba-9683-61bd584e13b9-dshm\") pod \"sub-test-model-94448f-kserve-865dd6f457-4982z\" (UID: \"27e508f3-1d9d-46ba-9683-61bd584e13b9\") " pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"
Jul 07 07:08:56.157504 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.157401    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wscfx\" (UniqueName: \"kubernetes.io/projected/27e508f3-1d9d-46ba-9683-61bd584e13b9-kube-api-access-wscfx\") pod \"sub-test-model-94448f-kserve-865dd6f457-4982z\" (UID: \"27e508f3-1d9d-46ba-9683-61bd584e13b9\") " pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"
Jul 07 07:08:56.157504 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.157445    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27e508f3-1d9d-46ba-9683-61bd584e13b9-model-cache\") pod \"sub-test-model-94448f-kserve-865dd6f457-4982z\" (UID: \"27e508f3-1d9d-46ba-9683-61bd584e13b9\") " pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"
Jul 07 07:08:56.258252 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.258229    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27e508f3-1d9d-46ba-9683-61bd584e13b9-model-cache\") pod \"sub-test-model-94448f-kserve-865dd6f457-4982z\" (UID: \"27e508f3-1d9d-46ba-9683-61bd584e13b9\") " pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"
Jul 07 07:08:56.258421 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.258275    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27e508f3-1d9d-46ba-9683-61bd584e13b9-tmp-dir\") pod \"sub-test-model-94448f-kserve-865dd6f457-4982z\" (UID: \"27e508f3-1d9d-46ba-9683-61bd584e13b9\") " pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"
Jul 07 07:08:56.258421 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.258297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27e508f3-1d9d-46ba-9683-61bd584e13b9-tls-certs\") pod \"sub-test-model-94448f-kserve-865dd6f457-4982z\" (UID: \"27e508f3-1d9d-46ba-9683-61bd584e13b9\") " pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"
Jul 07 07:08:56.258535 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.258436    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27e508f3-1d9d-46ba-9683-61bd584e13b9-home\") pod \"sub-test-model-94448f-kserve-865dd6f457-4982z\" (UID: \"27e508f3-1d9d-46ba-9683-61bd584e13b9\") " pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"
Jul 07 07:08:56.258535 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.258474    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27e508f3-1d9d-46ba-9683-61bd584e13b9-dshm\") pod \"sub-test-model-94448f-kserve-865dd6f457-4982z\" (UID: \"27e508f3-1d9d-46ba-9683-61bd584e13b9\") " pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"
Jul 07 07:08:56.258683 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.258592    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wscfx\" (UniqueName: \"kubernetes.io/projected/27e508f3-1d9d-46ba-9683-61bd584e13b9-kube-api-access-wscfx\") pod \"sub-test-model-94448f-kserve-865dd6f457-4982z\" (UID: \"27e508f3-1d9d-46ba-9683-61bd584e13b9\") " pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"
Jul 07 07:08:56.258683 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.258666    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27e508f3-1d9d-46ba-9683-61bd584e13b9-model-cache\") pod \"sub-test-model-94448f-kserve-865dd6f457-4982z\" (UID: \"27e508f3-1d9d-46ba-9683-61bd584e13b9\") " pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"
Jul 07 07:08:56.258798 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.258724    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27e508f3-1d9d-46ba-9683-61bd584e13b9-tmp-dir\") pod \"sub-test-model-94448f-kserve-865dd6f457-4982z\" (UID: \"27e508f3-1d9d-46ba-9683-61bd584e13b9\") " pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"
Jul 07 07:08:56.258884 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.258862    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27e508f3-1d9d-46ba-9683-61bd584e13b9-home\") pod \"sub-test-model-94448f-kserve-865dd6f457-4982z\" (UID: \"27e508f3-1d9d-46ba-9683-61bd584e13b9\") " pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"
Jul 07 07:08:56.260716 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.260692    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27e508f3-1d9d-46ba-9683-61bd584e13b9-dshm\") pod \"sub-test-model-94448f-kserve-865dd6f457-4982z\" (UID: \"27e508f3-1d9d-46ba-9683-61bd584e13b9\") " pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"
Jul 07 07:08:56.260941 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.260925    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27e508f3-1d9d-46ba-9683-61bd584e13b9-tls-certs\") pod \"sub-test-model-94448f-kserve-865dd6f457-4982z\" (UID: \"27e508f3-1d9d-46ba-9683-61bd584e13b9\") " pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"
Jul 07 07:08:56.267110 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.267092    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wscfx\" (UniqueName: \"kubernetes.io/projected/27e508f3-1d9d-46ba-9683-61bd584e13b9-kube-api-access-wscfx\") pod \"sub-test-model-94448f-kserve-865dd6f457-4982z\" (UID: \"27e508f3-1d9d-46ba-9683-61bd584e13b9\") " pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"
Jul 07 07:08:56.445067 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.444986    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"
Jul 07 07:08:56.578420 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:56.578395    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"]
Jul 07 07:08:56.580419 ip-10-0-133-92 kubenswrapper[2563]: W0707 07:08:56.580383    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27e508f3_1d9d_46ba_9683_61bd584e13b9.slice/crio-c02ebcbacf90a849546e73df5004a7e53e793a9e31ae12f8254c830e787201ec WatchSource:0}: Error finding container c02ebcbacf90a849546e73df5004a7e53e793a9e31ae12f8254c830e787201ec: Status 404 returned error can't find the container with id c02ebcbacf90a849546e73df5004a7e53e793a9e31ae12f8254c830e787201ec
Jul 07 07:08:57.263845 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:57.263809    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z" event={"ID":"27e508f3-1d9d-46ba-9683-61bd584e13b9","Type":"ContainerStarted","Data":"1ca28c332fb3b8264b71b92f97ab7078b31df19756f42fd82191920997cfd5f4"}
Jul 07 07:08:57.263845 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:57.263848    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z" event={"ID":"27e508f3-1d9d-46ba-9683-61bd584e13b9","Type":"ContainerStarted","Data":"c02ebcbacf90a849546e73df5004a7e53e793a9e31ae12f8254c830e787201ec"}
Jul 07 07:08:57.308971 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:08:57.308926    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z" podStartSLOduration=1.308911116 podStartE2EDuration="1.308911116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:08:57.306559306 +0000 UTC m=+2220.670755851" watchObservedRunningTime="2026-07-07 07:08:57.308911116 +0000 UTC m=+2220.673107660"
Jul 07 07:09:06.446076 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:06.446042    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"
Jul 07 07:09:06.446554 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:06.446099    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"
Jul 07 07:09:06.458514 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:06.458493    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"
Jul 07 07:09:07.323154 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:07.323126    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"
Jul 07 07:09:53.800733 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:53.800697    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"]
Jul 07 07:09:53.804104 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:53.804089    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"
Jul 07 07:09:53.806737 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:53.806714    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c72c9a\"/\"kube-root-ca.crt\""
Jul 07 07:09:53.807551 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:53.807527    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c72c9a\"/\"openshift-service-ca.crt\""
Jul 07 07:09:53.807685 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:53.807572    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c72c9a\"/\"default-dockercfg-m5thp\""
Jul 07 07:09:53.807685 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:53.807598    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c72c9a\"/\"sub-test-model-c72c9a-kserve-self-signed-certs\""
Jul 07 07:09:53.812959 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:53.812940    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"]
Jul 07 07:09:53.937656 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:53.937613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-tls-certs\") pod \"sub-test-model-c72c9a-kserve-d4c467cc-h572m\" (UID: \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\") " pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"
Jul 07 07:09:53.937656 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:53.937653    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-model-cache\") pod \"sub-test-model-c72c9a-kserve-d4c467cc-h572m\" (UID: \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\") " pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"
Jul 07 07:09:53.937813 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:53.937690    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-home\") pod \"sub-test-model-c72c9a-kserve-d4c467cc-h572m\" (UID: \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\") " pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"
Jul 07 07:09:53.937813 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:53.937705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-dshm\") pod \"sub-test-model-c72c9a-kserve-d4c467cc-h572m\" (UID: \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\") " pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"
Jul 07 07:09:53.937813 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:53.937787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-tmp-dir\") pod \"sub-test-model-c72c9a-kserve-d4c467cc-h572m\" (UID: \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\") " pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"
Jul 07 07:09:53.937910 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:53.937817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drgg7\" (UniqueName: \"kubernetes.io/projected/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-kube-api-access-drgg7\") pod \"sub-test-model-c72c9a-kserve-d4c467cc-h572m\" (UID: \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\") " pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"
Jul 07 07:09:54.038376 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:54.038354    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-tmp-dir\") pod \"sub-test-model-c72c9a-kserve-d4c467cc-h572m\" (UID: \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\") " pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"
Jul 07 07:09:54.038510 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:54.038388    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drgg7\" (UniqueName: \"kubernetes.io/projected/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-kube-api-access-drgg7\") pod \"sub-test-model-c72c9a-kserve-d4c467cc-h572m\" (UID: \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\") " pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"
Jul 07 07:09:54.038510 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:54.038441    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-tls-certs\") pod \"sub-test-model-c72c9a-kserve-d4c467cc-h572m\" (UID: \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\") " pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"
Jul 07 07:09:54.038510 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:54.038458    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-model-cache\") pod \"sub-test-model-c72c9a-kserve-d4c467cc-h572m\" (UID: \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\") " pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"
Jul 07 07:09:54.038510 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:54.038474    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-home\") pod \"sub-test-model-c72c9a-kserve-d4c467cc-h572m\" (UID: \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\") " pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"
Jul 07 07:09:54.038510 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:54.038494    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-dshm\") pod \"sub-test-model-c72c9a-kserve-d4c467cc-h572m\" (UID: \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\") " pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"
Jul 07 07:09:54.038829 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:54.038782    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-tmp-dir\") pod \"sub-test-model-c72c9a-kserve-d4c467cc-h572m\" (UID: \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\") " pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"
Jul 07 07:09:54.038895 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:54.038861    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-model-cache\") pod \"sub-test-model-c72c9a-kserve-d4c467cc-h572m\" (UID: \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\") " pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"
Jul 07 07:09:54.038950 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:54.038901    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-home\") pod \"sub-test-model-c72c9a-kserve-d4c467cc-h572m\" (UID: \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\") " pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"
Jul 07 07:09:54.040904 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:54.040887    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-dshm\") pod \"sub-test-model-c72c9a-kserve-d4c467cc-h572m\" (UID: \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\") " pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"
Jul 07 07:09:54.041177 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:54.041157    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-tls-certs\") pod \"sub-test-model-c72c9a-kserve-d4c467cc-h572m\" (UID: \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\") " pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"
Jul 07 07:09:54.047485 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:54.047456    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drgg7\" (UniqueName: \"kubernetes.io/projected/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-kube-api-access-drgg7\") pod \"sub-test-model-c72c9a-kserve-d4c467cc-h572m\" (UID: \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\") " pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"
Jul 07 07:09:54.114190 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:54.114135    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"
Jul 07 07:09:54.248504 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:54.248476    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"]
Jul 07 07:09:54.252074 ip-10-0-133-92 kubenswrapper[2563]: W0707 07:09:54.251757    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod455ce8fb_21cd_4121_9ce9_d2cd9efe4ed2.slice/crio-854e6c9595a3e7b5aac73497a1811f51a323a9f7fa51edd6ff7e73ac47ac74b6 WatchSource:0}: Error finding container 854e6c9595a3e7b5aac73497a1811f51a323a9f7fa51edd6ff7e73ac47ac74b6: Status 404 returned error can't find the container with id 854e6c9595a3e7b5aac73497a1811f51a323a9f7fa51edd6ff7e73ac47ac74b6
Jul 07 07:09:54.525007 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:54.524932    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m" event={"ID":"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2","Type":"ContainerStarted","Data":"a48727347eb67ae2b07b66a9eb55565cd0ebc1aae69989992fbf306d27ad4ee8"}
Jul 07 07:09:54.525007 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:54.524972    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m" event={"ID":"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2","Type":"ContainerStarted","Data":"854e6c9595a3e7b5aac73497a1811f51a323a9f7fa51edd6ff7e73ac47ac74b6"}
Jul 07 07:09:54.543226 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:09:54.543171    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m" podStartSLOduration=1.543157216 podStartE2EDuration="1.543157216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:09:54.541972743 +0000 UTC m=+2277.906169287" watchObservedRunningTime="2026-07-07 07:09:54.543157216 +0000 UTC m=+2277.907353761"
Jul 07 07:10:04.114876 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:04.114844    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"
Jul 07 07:10:04.115436 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:04.114891    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"
Jul 07 07:10:04.127375 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:04.127352    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"
Jul 07 07:10:04.580394 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:04.580364    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"
Jul 07 07:10:28.137101 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:28.137068    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"]
Jul 07 07:10:28.919799 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:28.919764    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"]
Jul 07 07:10:43.156467 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.156431    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z" podUID="27e508f3-1d9d-46ba-9683-61bd584e13b9" containerName="main" containerID="cri-o://1ca28c332fb3b8264b71b92f97ab7078b31df19756f42fd82191920997cfd5f4" gracePeriod=45
Jul 07 07:10:43.411734 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.411682    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"
Jul 07 07:10:43.518860 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.518828    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27e508f3-1d9d-46ba-9683-61bd584e13b9-tls-certs\") pod \"27e508f3-1d9d-46ba-9683-61bd584e13b9\" (UID: \"27e508f3-1d9d-46ba-9683-61bd584e13b9\") "
Jul 07 07:10:43.519002 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.518898    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27e508f3-1d9d-46ba-9683-61bd584e13b9-home\") pod \"27e508f3-1d9d-46ba-9683-61bd584e13b9\" (UID: \"27e508f3-1d9d-46ba-9683-61bd584e13b9\") "
Jul 07 07:10:43.519069 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.519006    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wscfx\" (UniqueName: \"kubernetes.io/projected/27e508f3-1d9d-46ba-9683-61bd584e13b9-kube-api-access-wscfx\") pod \"27e508f3-1d9d-46ba-9683-61bd584e13b9\" (UID: \"27e508f3-1d9d-46ba-9683-61bd584e13b9\") "
Jul 07 07:10:43.519137 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.519089    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27e508f3-1d9d-46ba-9683-61bd584e13b9-tmp-dir\") pod \"27e508f3-1d9d-46ba-9683-61bd584e13b9\" (UID: \"27e508f3-1d9d-46ba-9683-61bd584e13b9\") "
Jul 07 07:10:43.519137 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.519118    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27e508f3-1d9d-46ba-9683-61bd584e13b9-dshm\") pod \"27e508f3-1d9d-46ba-9683-61bd584e13b9\" (UID: \"27e508f3-1d9d-46ba-9683-61bd584e13b9\") "
Jul 07 07:10:43.519229 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.519141    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27e508f3-1d9d-46ba-9683-61bd584e13b9-model-cache\") pod \"27e508f3-1d9d-46ba-9683-61bd584e13b9\" (UID: \"27e508f3-1d9d-46ba-9683-61bd584e13b9\") "
Jul 07 07:10:43.519229 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.519091    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e508f3-1d9d-46ba-9683-61bd584e13b9-home" (OuterVolumeSpecName: "home") pod "27e508f3-1d9d-46ba-9683-61bd584e13b9" (UID: "27e508f3-1d9d-46ba-9683-61bd584e13b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:10:43.519399 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.519372    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e508f3-1d9d-46ba-9683-61bd584e13b9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "27e508f3-1d9d-46ba-9683-61bd584e13b9" (UID: "27e508f3-1d9d-46ba-9683-61bd584e13b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:10:43.519399 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.519381    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e508f3-1d9d-46ba-9683-61bd584e13b9-model-cache" (OuterVolumeSpecName: "model-cache") pod "27e508f3-1d9d-46ba-9683-61bd584e13b9" (UID: "27e508f3-1d9d-46ba-9683-61bd584e13b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:10:43.519784 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.519467    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27e508f3-1d9d-46ba-9683-61bd584e13b9-home\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:10:43.519784 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.519483    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27e508f3-1d9d-46ba-9683-61bd584e13b9-tmp-dir\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:10:43.519784 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.519494    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27e508f3-1d9d-46ba-9683-61bd584e13b9-model-cache\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:10:43.521011 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.520990    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27e508f3-1d9d-46ba-9683-61bd584e13b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27e508f3-1d9d-46ba-9683-61bd584e13b9" (UID: "27e508f3-1d9d-46ba-9683-61bd584e13b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 07:10:43.521105 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.521087    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e508f3-1d9d-46ba-9683-61bd584e13b9-kube-api-access-wscfx" (OuterVolumeSpecName: "kube-api-access-wscfx") pod "27e508f3-1d9d-46ba-9683-61bd584e13b9" (UID: "27e508f3-1d9d-46ba-9683-61bd584e13b9"). InnerVolumeSpecName "kube-api-access-wscfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 07:10:43.521167 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.521146    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e508f3-1d9d-46ba-9683-61bd584e13b9-dshm" (OuterVolumeSpecName: "dshm") pod "27e508f3-1d9d-46ba-9683-61bd584e13b9" (UID: "27e508f3-1d9d-46ba-9683-61bd584e13b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:10:43.620862 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.620837    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wscfx\" (UniqueName: \"kubernetes.io/projected/27e508f3-1d9d-46ba-9683-61bd584e13b9-kube-api-access-wscfx\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:10:43.620862 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.620860    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27e508f3-1d9d-46ba-9683-61bd584e13b9-dshm\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:10:43.620995 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.620873    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27e508f3-1d9d-46ba-9683-61bd584e13b9-tls-certs\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:10:43.747312 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.747255    2563 generic.go:356] "Generic (PLEG): container finished" podID="27e508f3-1d9d-46ba-9683-61bd584e13b9" containerID="1ca28c332fb3b8264b71b92f97ab7078b31df19756f42fd82191920997cfd5f4" exitCode=0
Jul 07 07:10:43.747312 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.747296    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"
Jul 07 07:10:43.747430 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.747347    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z" event={"ID":"27e508f3-1d9d-46ba-9683-61bd584e13b9","Type":"ContainerDied","Data":"1ca28c332fb3b8264b71b92f97ab7078b31df19756f42fd82191920997cfd5f4"}
Jul 07 07:10:43.747430 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.747385    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z" event={"ID":"27e508f3-1d9d-46ba-9683-61bd584e13b9","Type":"ContainerDied","Data":"c02ebcbacf90a849546e73df5004a7e53e793a9e31ae12f8254c830e787201ec"}
Jul 07 07:10:43.747430 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.747402    2563 scope.go:117] "RemoveContainer" containerID="1ca28c332fb3b8264b71b92f97ab7078b31df19756f42fd82191920997cfd5f4"
Jul 07 07:10:43.756087 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.756070    2563 scope.go:117] "RemoveContainer" containerID="1ca28c332fb3b8264b71b92f97ab7078b31df19756f42fd82191920997cfd5f4"
Jul 07 07:10:43.756311 ip-10-0-133-92 kubenswrapper[2563]: E0707 07:10:43.756294    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ca28c332fb3b8264b71b92f97ab7078b31df19756f42fd82191920997cfd5f4\": container with ID starting with 1ca28c332fb3b8264b71b92f97ab7078b31df19756f42fd82191920997cfd5f4 not found: ID does not exist" containerID="1ca28c332fb3b8264b71b92f97ab7078b31df19756f42fd82191920997cfd5f4"
Jul 07 07:10:43.756369 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.756317    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca28c332fb3b8264b71b92f97ab7078b31df19756f42fd82191920997cfd5f4"} err="failed to get container status \"1ca28c332fb3b8264b71b92f97ab7078b31df19756f42fd82191920997cfd5f4\": rpc error: code = NotFound desc = could not find container \"1ca28c332fb3b8264b71b92f97ab7078b31df19756f42fd82191920997cfd5f4\": container with ID starting with 1ca28c332fb3b8264b71b92f97ab7078b31df19756f42fd82191920997cfd5f4 not found: ID does not exist"
Jul 07 07:10:43.767547 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.767528    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"]
Jul 07 07:10:43.773772 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.773753    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-94448f/sub-test-model-94448f-kserve-865dd6f457-4982z"]
Jul 07 07:10:43.939511 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:43.939482    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m" podUID="455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2" containerName="main" containerID="cri-o://a48727347eb67ae2b07b66a9eb55565cd0ebc1aae69989992fbf306d27ad4ee8" gracePeriod=45
Jul 07 07:10:44.197080 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.197061    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"
Jul 07 07:10:44.326967 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.326901    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-home\") pod \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\" (UID: \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\") "
Jul 07 07:10:44.326967 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.326952    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-model-cache\") pod \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\" (UID: \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\") "
Jul 07 07:10:44.327147 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.326972    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-dshm\") pod \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\" (UID: \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\") "
Jul 07 07:10:44.327147 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.327013    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drgg7\" (UniqueName: \"kubernetes.io/projected/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-kube-api-access-drgg7\") pod \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\" (UID: \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\") "
Jul 07 07:10:44.327147 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.327029    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-tls-certs\") pod \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\" (UID: \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\") "
Jul 07 07:10:44.327147 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.327047    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-tmp-dir\") pod \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\" (UID: \"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2\") "
Jul 07 07:10:44.327364 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.327191    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-home" (OuterVolumeSpecName: "home") pod "455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2" (UID: "455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:10:44.327364 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.327204    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-model-cache" (OuterVolumeSpecName: "model-cache") pod "455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2" (UID: "455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:10:44.327474 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.327416    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2" (UID: "455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:10:44.327474 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.327432    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-home\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:10:44.327474 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.327451    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-model-cache\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:10:44.329279 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.329256    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-dshm" (OuterVolumeSpecName: "dshm") pod "455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2" (UID: "455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:10:44.329408 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.329321    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2" (UID: "455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 07:10:44.329470 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.329413    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-kube-api-access-drgg7" (OuterVolumeSpecName: "kube-api-access-drgg7") pod "455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2" (UID: "455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2"). InnerVolumeSpecName "kube-api-access-drgg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 07:10:44.428134 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.428110    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drgg7\" (UniqueName: \"kubernetes.io/projected/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-kube-api-access-drgg7\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:10:44.428134 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.428130    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-tls-certs\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:10:44.428308 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.428141    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-tmp-dir\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:10:44.428308 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.428150    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2-dshm\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:10:44.752344 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.752275    2563 generic.go:356] "Generic (PLEG): container finished" podID="455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2" containerID="a48727347eb67ae2b07b66a9eb55565cd0ebc1aae69989992fbf306d27ad4ee8" exitCode=0
Jul 07 07:10:44.752344 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.752319    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"
Jul 07 07:10:44.752521 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.752363    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m" event={"ID":"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2","Type":"ContainerDied","Data":"a48727347eb67ae2b07b66a9eb55565cd0ebc1aae69989992fbf306d27ad4ee8"}
Jul 07 07:10:44.752521 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.752397    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m" event={"ID":"455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2","Type":"ContainerDied","Data":"854e6c9595a3e7b5aac73497a1811f51a323a9f7fa51edd6ff7e73ac47ac74b6"}
Jul 07 07:10:44.752521 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.752414    2563 scope.go:117] "RemoveContainer" containerID="a48727347eb67ae2b07b66a9eb55565cd0ebc1aae69989992fbf306d27ad4ee8"
Jul 07 07:10:44.761353 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.761340    2563 scope.go:117] "RemoveContainer" containerID="a48727347eb67ae2b07b66a9eb55565cd0ebc1aae69989992fbf306d27ad4ee8"
Jul 07 07:10:44.761561 ip-10-0-133-92 kubenswrapper[2563]: E0707 07:10:44.761542    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a48727347eb67ae2b07b66a9eb55565cd0ebc1aae69989992fbf306d27ad4ee8\": container with ID starting with a48727347eb67ae2b07b66a9eb55565cd0ebc1aae69989992fbf306d27ad4ee8 not found: ID does not exist" containerID="a48727347eb67ae2b07b66a9eb55565cd0ebc1aae69989992fbf306d27ad4ee8"
Jul 07 07:10:44.761610 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.761567    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a48727347eb67ae2b07b66a9eb55565cd0ebc1aae69989992fbf306d27ad4ee8"} err="failed to get container status \"a48727347eb67ae2b07b66a9eb55565cd0ebc1aae69989992fbf306d27ad4ee8\": rpc error: code = NotFound desc = could not find container \"a48727347eb67ae2b07b66a9eb55565cd0ebc1aae69989992fbf306d27ad4ee8\": container with ID starting with a48727347eb67ae2b07b66a9eb55565cd0ebc1aae69989992fbf306d27ad4ee8 not found: ID does not exist"
Jul 07 07:10:44.773741 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.773720    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"]
Jul 07 07:10:44.777530 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:44.777506    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c72c9a/sub-test-model-c72c9a-kserve-d4c467cc-h572m"]
Jul 07 07:10:45.171550 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:45.171523    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27e508f3-1d9d-46ba-9683-61bd584e13b9" path="/var/lib/kubelet/pods/27e508f3-1d9d-46ba-9683-61bd584e13b9/volumes"
Jul 07 07:10:45.171912 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:10:45.171897    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2" path="/var/lib/kubelet/pods/455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2/volumes"
Jul 07 07:11:15.195801 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.195773    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"]
Jul 07 07:11:15.196252 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.196100    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27e508f3-1d9d-46ba-9683-61bd584e13b9" containerName="main"
Jul 07 07:11:15.196252 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.196110    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="27e508f3-1d9d-46ba-9683-61bd584e13b9" containerName="main"
Jul 07 07:11:15.196252 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.196126    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2" containerName="main"
Jul 07 07:11:15.196252 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.196131    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2" containerName="main"
Jul 07 07:11:15.196252 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.196189    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="27e508f3-1d9d-46ba-9683-61bd584e13b9" containerName="main"
Jul 07 07:11:15.196252 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.196198    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="455ce8fb-21cd-4121-9ce9-d2cd9efe4ed2" containerName="main"
Jul 07 07:11:15.199113 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.199097    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"
Jul 07 07:11:15.201490 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.201467    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c72c9a\"/\"openshift-service-ca.crt\""
Jul 07 07:11:15.201972 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.201954    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c72c9a\"/\"default-dockercfg-m5thp\""
Jul 07 07:11:15.201972 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.201962    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c72c9a\"/\"rate-test-model-c72c9a-kserve-self-signed-certs\""
Jul 07 07:11:15.202155 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.202013    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c72c9a\"/\"kube-root-ca.crt\""
Jul 07 07:11:15.210837 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.210819    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"]
Jul 07 07:11:15.253651 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.253609    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-model-cache\") pod \"rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s\" (UID: \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\") " pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"
Jul 07 07:11:15.253739 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.253651    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-home\") pod \"rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s\" (UID: \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\") " pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"
Jul 07 07:11:15.253739 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.253690    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-tls-certs\") pod \"rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s\" (UID: \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\") " pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"
Jul 07 07:11:15.253739 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.253719    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-tmp-dir\") pod \"rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s\" (UID: \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\") " pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"
Jul 07 07:11:15.253739 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.253737    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv5s8\" (UniqueName: \"kubernetes.io/projected/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-kube-api-access-hv5s8\") pod \"rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s\" (UID: \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\") " pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"
Jul 07 07:11:15.253876 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.253797    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-dshm\") pod \"rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s\" (UID: \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\") " pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"
Jul 07 07:11:15.354847 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.354819    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-model-cache\") pod \"rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s\" (UID: \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\") " pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"
Jul 07 07:11:15.354937 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.354861    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-home\") pod \"rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s\" (UID: \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\") " pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"
Jul 07 07:11:15.354937 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.354895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-tls-certs\") pod \"rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s\" (UID: \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\") " pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"
Jul 07 07:11:15.355050 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.354938    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-tmp-dir\") pod \"rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s\" (UID: \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\") " pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"
Jul 07 07:11:15.355050 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.354966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv5s8\" (UniqueName: \"kubernetes.io/projected/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-kube-api-access-hv5s8\") pod \"rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s\" (UID: \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\") " pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"
Jul 07 07:11:15.355050 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.354993    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-dshm\") pod \"rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s\" (UID: \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\") " pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"
Jul 07 07:11:15.355278 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.355256    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-model-cache\") pod \"rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s\" (UID: \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\") " pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"
Jul 07 07:11:15.355350 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.355289    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-home\") pod \"rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s\" (UID: \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\") " pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"
Jul 07 07:11:15.355350 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.355311    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-tmp-dir\") pod \"rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s\" (UID: \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\") " pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"
Jul 07 07:11:15.357232 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.357212    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-dshm\") pod \"rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s\" (UID: \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\") " pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"
Jul 07 07:11:15.357489 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.357470    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-tls-certs\") pod \"rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s\" (UID: \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\") " pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"
Jul 07 07:11:15.363387 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.363367    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv5s8\" (UniqueName: \"kubernetes.io/projected/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-kube-api-access-hv5s8\") pod \"rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s\" (UID: \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\") " pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"
Jul 07 07:11:15.508075 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.508022    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"
Jul 07 07:11:15.636188 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.636164    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"]
Jul 07 07:11:15.638443 ip-10-0-133-92 kubenswrapper[2563]: W0707 07:11:15.638418    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc59a6d9f_0b74_4596_b82d_7fcc5e8a1e48.slice/crio-919d093937b98ad6725baf3a7d4e5a4adb684e47005ec2aa9e5768bb5fe7d608 WatchSource:0}: Error finding container 919d093937b98ad6725baf3a7d4e5a4adb684e47005ec2aa9e5768bb5fe7d608: Status 404 returned error can't find the container with id 919d093937b98ad6725baf3a7d4e5a4adb684e47005ec2aa9e5768bb5fe7d608
Jul 07 07:11:15.892186 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.892147    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s" event={"ID":"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48","Type":"ContainerStarted","Data":"abec21b726701a4ff82b9dd41f8e7a2612ff9a077847bf70129252708130b264"}
Jul 07 07:11:15.892352 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:15.892194    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s" event={"ID":"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48","Type":"ContainerStarted","Data":"919d093937b98ad6725baf3a7d4e5a4adb684e47005ec2aa9e5768bb5fe7d608"}
Jul 07 07:11:25.509092 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:25.509057    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"
Jul 07 07:11:25.509556 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:25.509110    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"
Jul 07 07:11:25.521468 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:25.521443    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"
Jul 07 07:11:25.542240 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:25.542188    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s" podStartSLOduration=10.542169526 podStartE2EDuration="10.542169526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 07:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:11:15.912923306 +0000 UTC m=+2359.277119852" watchObservedRunningTime="2026-07-07 07:11:25.542169526 +0000 UTC m=+2368.906366071"
Jul 07 07:11:25.949974 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:25.949949    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"
Jul 07 07:11:58.998030 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:11:58.997975    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"]
Jul 07 07:12:14.018616 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:14.018576    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s" podUID="c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48" containerName="main" containerID="cri-o://abec21b726701a4ff82b9dd41f8e7a2612ff9a077847bf70129252708130b264" gracePeriod=45
Jul 07 07:12:14.161988 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:14.161956    2563 generic.go:356] "Generic (PLEG): container finished" podID="c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48" containerID="abec21b726701a4ff82b9dd41f8e7a2612ff9a077847bf70129252708130b264" exitCode=0
Jul 07 07:12:14.162150 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:14.162032    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s" event={"ID":"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48","Type":"ContainerDied","Data":"abec21b726701a4ff82b9dd41f8e7a2612ff9a077847bf70129252708130b264"}
Jul 07 07:12:14.272700 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:14.270806    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"
Jul 07 07:12:14.346143 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:14.346115    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hv5s8\" (UniqueName: \"kubernetes.io/projected/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-kube-api-access-hv5s8\") pod \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\" (UID: \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\") "
Jul 07 07:12:14.346323 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:14.346167    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-tmp-dir\") pod \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\" (UID: \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\") "
Jul 07 07:12:14.346323 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:14.346202    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-model-cache\") pod \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\" (UID: \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\") "
Jul 07 07:12:14.346323 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:14.346225    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-home\") pod \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\" (UID: \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\") "
Jul 07 07:12:14.346323 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:14.346250    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-dshm\") pod \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\" (UID: \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\") "
Jul 07 07:12:14.346323 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:14.346308    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-tls-certs\") pod \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\" (UID: \"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48\") "
Jul 07 07:12:14.346599 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:14.346517    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-model-cache" (OuterVolumeSpecName: "model-cache") pod "c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48" (UID: "c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:12:14.346599 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:14.346531    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-home" (OuterVolumeSpecName: "home") pod "c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48" (UID: "c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:12:14.346599 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:14.346541    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48" (UID: "c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:12:14.348433 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:14.348410    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-dshm" (OuterVolumeSpecName: "dshm") pod "c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48" (UID: "c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 07:12:14.348538 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:14.348434    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-kube-api-access-hv5s8" (OuterVolumeSpecName: "kube-api-access-hv5s8") pod "c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48" (UID: "c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48"). InnerVolumeSpecName "kube-api-access-hv5s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 07:12:14.348538 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:14.348460    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48" (UID: "c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 07:12:14.446866 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:14.446844    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-tmp-dir\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:12:14.446866 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:14.446865    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-model-cache\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:12:14.446994 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:14.446873    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-home\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:12:14.446994 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:14.446880    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-dshm\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:12:14.446994 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:14.446888    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-tls-certs\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:12:14.446994 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:14.446896    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv5s8\" (UniqueName: \"kubernetes.io/projected/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48-kube-api-access-hv5s8\") on node \"ip-10-0-133-92.ec2.internal\" DevicePath \"\""
Jul 07 07:12:15.167884 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:15.167860    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"
Jul 07 07:12:15.171369 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:15.171331    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s" event={"ID":"c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48","Type":"ContainerDied","Data":"919d093937b98ad6725baf3a7d4e5a4adb684e47005ec2aa9e5768bb5fe7d608"}
Jul 07 07:12:15.171520 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:15.171382    2563 scope.go:117] "RemoveContainer" containerID="abec21b726701a4ff82b9dd41f8e7a2612ff9a077847bf70129252708130b264"
Jul 07 07:12:15.191274 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:15.191253    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"]
Jul 07 07:12:15.193627 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:15.193606    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c72c9a/rate-test-model-c72c9a-kserve-6d68d6fbc9-rnx6s"]
Jul 07 07:12:17.172554 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:17.172456    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48" path="/var/lib/kubelet/pods/c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48/volumes"
Jul 07 07:12:56.046525 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:56.046478    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-6hmxm_c21ff8a6-6973-47f7-a056-9d59f933a8df/manager/0.log"
Jul 07 07:12:56.768058 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:56.768031    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c66c8b568-dwbks_3de2d723-cc06-48f1-a5cb-28994950a44b/copy-manifests/0.log"
Jul 07 07:12:56.774811 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:56.774776    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c66c8b568-dwbks_3de2d723-cc06-48f1-a5cb-28994950a44b/manager/0.log"
Jul 07 07:12:57.016822 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:57.016786    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-b6stw_0d222fa8-990f-4c76-b79a-abbca96a5cdd/postgres/0.log"
Jul 07 07:12:58.639735 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:58.639705    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-zzcpc_fcdc7918-ea72-4e20-b93d-e41e6c86480e/kuadrant-console-plugin/0.log"
Jul 07 07:12:59.826938 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:12:59.826913    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74994db979-jwcx7_ce07d74d-6b0a-4e94-b31c-c9553c5f3a1b/kube-auth-proxy/0.log"
Jul 07 07:13:00.069388 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:00.069344    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-rklkm_be371c12-0c30-4424-8f5d-ac6a5883f509/payload-pre-processing/0.log"
Jul 07 07:13:00.187951 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:00.187867    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-lk6nm_28572129-d52e-4589-9993-2adf95c458e2/payload-processing/0.log"
Jul 07 07:13:00.882820 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:00.882791    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-fm56g_fe3b8395-7fde-4c15-85d4-b7ff5769ea14/main/0.log"
Jul 07 07:13:01.130613 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:01.130580    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-gmp7z_55ae6c62-626a-49ad-83f8-db6844af3425/main/0.log"
Jul 07 07:13:01.257013 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:01.256919    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-wlqrz_ca5278a0-49dd-4b23-8677-0972b7bb97f3/main/0.log"
Jul 07 07:13:08.409196 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:08.409160    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-57vh8_ee9fd62f-81cf-4544-8454-9fa9f53c0768/global-pull-secret-syncer/0.log"
Jul 07 07:13:08.567440 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:08.567406    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xh2bl_c5d1a464-df78-4f33-8ae7-f81183a9fb39/konnectivity-agent/0.log"
Jul 07 07:13:08.632799 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:08.632765    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-92.ec2.internal_9ed9ea7211a1e46135e9e4a3c3edc4de/haproxy/0.log"
Jul 07 07:13:13.115661 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:13.115612    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-zzcpc_fcdc7918-ea72-4e20-b93d-e41e6c86480e/kuadrant-console-plugin/0.log"
Jul 07 07:13:15.239878 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:15.239847    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-f7646d95-nsbd6_45ef3bb7-96c9-4d6c-8214-093409d1856e/metrics-server/0.log"
Jul 07 07:13:15.265776 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:15.265750    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-p5kjp_26aef36b-c5d2-4c16-9bfc-917f6eba2bb9/monitoring-plugin/0.log"
Jul 07 07:13:15.295071 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:15.295045    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2h7jr_a17275c3-428d-44ef-8207-0e1ea72271d4/node-exporter/0.log"
Jul 07 07:13:15.329260 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:15.329236    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2h7jr_a17275c3-428d-44ef-8207-0e1ea72271d4/kube-rbac-proxy/0.log"
Jul 07 07:13:15.353330 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:15.353306    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2h7jr_a17275c3-428d-44ef-8207-0e1ea72271d4/init-textfile/0.log"
Jul 07 07:13:15.605486 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:15.605459    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-9d268_c6669dd2-240f-4113-9b5e-0aaa41741cca/kube-rbac-proxy-main/0.log"
Jul 07 07:13:15.626699 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:15.626670    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-9d268_c6669dd2-240f-4113-9b5e-0aaa41741cca/kube-rbac-proxy-self/0.log"
Jul 07 07:13:15.645086 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:15.645064    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-9d268_c6669dd2-240f-4113-9b5e-0aaa41741cca/openshift-state-metrics/0.log"
Jul 07 07:13:15.687512 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:15.687490    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f6429ad-0f44-4eaa-9638-0627dc4e392a/prometheus/0.log"
Jul 07 07:13:15.708240 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:15.708215    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f6429ad-0f44-4eaa-9638-0627dc4e392a/config-reloader/0.log"
Jul 07 07:13:15.729063 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:15.729039    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f6429ad-0f44-4eaa-9638-0627dc4e392a/thanos-sidecar/0.log"
Jul 07 07:13:15.752489 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:15.752435    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f6429ad-0f44-4eaa-9638-0627dc4e392a/kube-rbac-proxy-web/0.log"
Jul 07 07:13:15.778195 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:15.778174    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f6429ad-0f44-4eaa-9638-0627dc4e392a/kube-rbac-proxy/0.log"
Jul 07 07:13:15.803035 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:15.803002    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f6429ad-0f44-4eaa-9638-0627dc4e392a/kube-rbac-proxy-thanos/0.log"
Jul 07 07:13:15.824879 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:15.824850    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f6429ad-0f44-4eaa-9638-0627dc4e392a/init-config-reloader/0.log"
Jul 07 07:13:15.861187 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:15.861164    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-kjcbn_c81bc8ba-1e33-4641-88fb-486835eaa393/prometheus-operator/0.log"
Jul 07 07:13:15.889901 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:15.889853    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-kjcbn_c81bc8ba-1e33-4641-88fb-486835eaa393/kube-rbac-proxy/0.log"
Jul 07 07:13:15.926109 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:15.926086    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-vwcc9_7f71831a-ee04-48df-a4f3-9636ab2da704/prometheus-operator-admission-webhook/0.log"
Jul 07 07:13:18.748646 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:18.748612    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5d577798c-n7ggq_52c18bf8-c9a6-45c1-a7a5-6458f15edc6b/console/0.log"
Jul 07 07:13:18.786955 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:18.786924    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f8h84/perf-node-gather-daemonset-btr2h"]
Jul 07 07:13:18.787280 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:18.787268    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48" containerName="main"
Jul 07 07:13:18.787347 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:18.787281    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48" containerName="main"
Jul 07 07:13:18.787388 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:18.787353    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c59a6d9f-0b74-4596-b82d-7fcc5e8a1e48" containerName="main"
Jul 07 07:13:18.790119 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:18.790102    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f8h84/perf-node-gather-daemonset-btr2h"
Jul 07 07:13:18.795884 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:18.795863    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f8h84\"/\"kube-root-ca.crt\""
Jul 07 07:13:18.795993 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:18.795884    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-f8h84\"/\"default-dockercfg-xq6kg\""
Jul 07 07:13:18.796469 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:18.796376    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f8h84\"/\"openshift-service-ca.crt\""
Jul 07 07:13:18.812685 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:18.812662    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f8h84/perf-node-gather-daemonset-btr2h"]
Jul 07 07:13:18.926786 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:18.926755    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4566b9f0-5810-4c46-a991-6dea2fd4654c-lib-modules\") pod \"perf-node-gather-daemonset-btr2h\" (UID: \"4566b9f0-5810-4c46-a991-6dea2fd4654c\") " pod="openshift-must-gather-f8h84/perf-node-gather-daemonset-btr2h"
Jul 07 07:13:18.926786 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:18.926784    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4566b9f0-5810-4c46-a991-6dea2fd4654c-proc\") pod \"perf-node-gather-daemonset-btr2h\" (UID: \"4566b9f0-5810-4c46-a991-6dea2fd4654c\") " pod="openshift-must-gather-f8h84/perf-node-gather-daemonset-btr2h"
Jul 07 07:13:18.926964 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:18.926823    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4566b9f0-5810-4c46-a991-6dea2fd4654c-podres\") pod \"perf-node-gather-daemonset-btr2h\" (UID: \"4566b9f0-5810-4c46-a991-6dea2fd4654c\") " pod="openshift-must-gather-f8h84/perf-node-gather-daemonset-btr2h"
Jul 07 07:13:18.926964 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:18.926910    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4566b9f0-5810-4c46-a991-6dea2fd4654c-sys\") pod \"perf-node-gather-daemonset-btr2h\" (UID: \"4566b9f0-5810-4c46-a991-6dea2fd4654c\") " pod="openshift-must-gather-f8h84/perf-node-gather-daemonset-btr2h"
Jul 07 07:13:18.926964 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:18.926947    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6mjz\" (UniqueName: \"kubernetes.io/projected/4566b9f0-5810-4c46-a991-6dea2fd4654c-kube-api-access-x6mjz\") pod \"perf-node-gather-daemonset-btr2h\" (UID: \"4566b9f0-5810-4c46-a991-6dea2fd4654c\") " pod="openshift-must-gather-f8h84/perf-node-gather-daemonset-btr2h"
Jul 07 07:13:19.028358 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:19.028281    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4566b9f0-5810-4c46-a991-6dea2fd4654c-podres\") pod \"perf-node-gather-daemonset-btr2h\" (UID: \"4566b9f0-5810-4c46-a991-6dea2fd4654c\") " pod="openshift-must-gather-f8h84/perf-node-gather-daemonset-btr2h"
Jul 07 07:13:19.028358 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:19.028338    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4566b9f0-5810-4c46-a991-6dea2fd4654c-sys\") pod \"perf-node-gather-daemonset-btr2h\" (UID: \"4566b9f0-5810-4c46-a991-6dea2fd4654c\") " pod="openshift-must-gather-f8h84/perf-node-gather-daemonset-btr2h"
Jul 07 07:13:19.028528 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:19.028377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6mjz\" (UniqueName: \"kubernetes.io/projected/4566b9f0-5810-4c46-a991-6dea2fd4654c-kube-api-access-x6mjz\") pod \"perf-node-gather-daemonset-btr2h\" (UID: \"4566b9f0-5810-4c46-a991-6dea2fd4654c\") " pod="openshift-must-gather-f8h84/perf-node-gather-daemonset-btr2h"
Jul 07 07:13:19.028528 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:19.028464    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4566b9f0-5810-4c46-a991-6dea2fd4654c-podres\") pod \"perf-node-gather-daemonset-btr2h\" (UID: \"4566b9f0-5810-4c46-a991-6dea2fd4654c\") " pod="openshift-must-gather-f8h84/perf-node-gather-daemonset-btr2h"
Jul 07 07:13:19.028528 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:19.028470    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4566b9f0-5810-4c46-a991-6dea2fd4654c-sys\") pod \"perf-node-gather-daemonset-btr2h\" (UID: \"4566b9f0-5810-4c46-a991-6dea2fd4654c\") " pod="openshift-must-gather-f8h84/perf-node-gather-daemonset-btr2h"
Jul 07 07:13:19.028724 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:19.028529    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4566b9f0-5810-4c46-a991-6dea2fd4654c-lib-modules\") pod \"perf-node-gather-daemonset-btr2h\" (UID: \"4566b9f0-5810-4c46-a991-6dea2fd4654c\") " pod="openshift-must-gather-f8h84/perf-node-gather-daemonset-btr2h"
Jul 07 07:13:19.028724 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:19.028572    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4566b9f0-5810-4c46-a991-6dea2fd4654c-proc\") pod \"perf-node-gather-daemonset-btr2h\" (UID: \"4566b9f0-5810-4c46-a991-6dea2fd4654c\") " pod="openshift-must-gather-f8h84/perf-node-gather-daemonset-btr2h"
Jul 07 07:13:19.028724 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:19.028694    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4566b9f0-5810-4c46-a991-6dea2fd4654c-proc\") pod \"perf-node-gather-daemonset-btr2h\" (UID: \"4566b9f0-5810-4c46-a991-6dea2fd4654c\") " pod="openshift-must-gather-f8h84/perf-node-gather-daemonset-btr2h"
Jul 07 07:13:19.028724 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:19.028703    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4566b9f0-5810-4c46-a991-6dea2fd4654c-lib-modules\") pod \"perf-node-gather-daemonset-btr2h\" (UID: \"4566b9f0-5810-4c46-a991-6dea2fd4654c\") " pod="openshift-must-gather-f8h84/perf-node-gather-daemonset-btr2h"
Jul 07 07:13:19.037242 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:19.037224    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6mjz\" (UniqueName: \"kubernetes.io/projected/4566b9f0-5810-4c46-a991-6dea2fd4654c-kube-api-access-x6mjz\") pod \"perf-node-gather-daemonset-btr2h\" (UID: \"4566b9f0-5810-4c46-a991-6dea2fd4654c\") " pod="openshift-must-gather-f8h84/perf-node-gather-daemonset-btr2h"
Jul 07 07:13:19.099877 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:19.099842    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f8h84/perf-node-gather-daemonset-btr2h"
Jul 07 07:13:19.234193 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:19.234159    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f8h84/perf-node-gather-daemonset-btr2h"]
Jul 07 07:13:19.237259 ip-10-0-133-92 kubenswrapper[2563]: W0707 07:13:19.237231    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4566b9f0_5810_4c46_a991_6dea2fd4654c.slice/crio-5a883413e063e5764282985915b0f53492322f58f9f43eff3ff883050ef68cbc WatchSource:0}: Error finding container 5a883413e063e5764282985915b0f53492322f58f9f43eff3ff883050ef68cbc: Status 404 returned error can't find the container with id 5a883413e063e5764282985915b0f53492322f58f9f43eff3ff883050ef68cbc
Jul 07 07:13:19.453310 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:19.453277    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f8h84/perf-node-gather-daemonset-btr2h" event={"ID":"4566b9f0-5810-4c46-a991-6dea2fd4654c","Type":"ContainerStarted","Data":"9dce6edbeabd5cdbe4375487961b1c785fc7a7b9014492def695ba2292975cca"}
Jul 07 07:13:19.453503 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:19.453326    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f8h84/perf-node-gather-daemonset-btr2h" event={"ID":"4566b9f0-5810-4c46-a991-6dea2fd4654c","Type":"ContainerStarted","Data":"5a883413e063e5764282985915b0f53492322f58f9f43eff3ff883050ef68cbc"}
Jul 07 07:13:19.453503 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:19.453353    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-f8h84/perf-node-gather-daemonset-btr2h"
Jul 07 07:13:19.474735 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:19.474690    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-f8h84/perf-node-gather-daemonset-btr2h" podStartSLOduration=1.474675903 podStartE2EDuration="1.474675903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 07:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 07:13:19.472653094 +0000 UTC m=+2482.836849649" watchObservedRunningTime="2026-07-07 07:13:19.474675903 +0000 UTC m=+2482.838872455"
Jul 07 07:13:20.693054 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:20.693013    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gvkjl_c196cdf7-b0a5-4fa4-8f02-9176334b6340/dns/0.log"
Jul 07 07:13:20.721790 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:20.721761    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gvkjl_c196cdf7-b0a5-4fa4-8f02-9176334b6340/kube-rbac-proxy/0.log"
Jul 07 07:13:20.941070 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:20.941036    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mbqms_4aaa39b2-a25e-4a3c-b3b8-b65d0b6d140b/dns-node-resolver/0.log"
Jul 07 07:13:21.600524 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:21.600487    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jg66b_327fd7db-21be-45ef-9efd-6c3f46c1e663/node-ca/0.log"
Jul 07 07:13:22.812947 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:22.812919    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74994db979-jwcx7_ce07d74d-6b0a-4e94-b31c-c9553c5f3a1b/kube-auth-proxy/0.log"
Jul 07 07:13:22.910866 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:22.910835    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-rklkm_be371c12-0c30-4424-8f5d-ac6a5883f509/payload-pre-processing/0.log"
Jul 07 07:13:22.981019 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:22.980985    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-lk6nm_28572129-d52e-4589-9993-2adf95c458e2/payload-processing/0.log"
Jul 07 07:13:23.734729 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:23.734699    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qmc4r_0cc3267d-e426-4fa1-88c9-9fbdcda4e8ee/serve-healthcheck-canary/0.log"
Jul 07 07:13:24.359370 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:24.359338    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vhq4w_e5f0669f-f5b0-4d36-b791-681380b94eb6/kube-rbac-proxy/0.log"
Jul 07 07:13:24.381974 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:24.381951    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vhq4w_e5f0669f-f5b0-4d36-b791-681380b94eb6/exporter/0.log"
Jul 07 07:13:24.401743 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:24.401701    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vhq4w_e5f0669f-f5b0-4d36-b791-681380b94eb6/extractor/0.log"
Jul 07 07:13:25.469577 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:25.469549    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-f8h84/perf-node-gather-daemonset-btr2h"
Jul 07 07:13:26.700310 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:26.700281    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-6hmxm_c21ff8a6-6973-47f7-a056-9d59f933a8df/manager/0.log"
Jul 07 07:13:27.061534 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:27.061499    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c66c8b568-dwbks_3de2d723-cc06-48f1-a5cb-28994950a44b/manager/0.log"
Jul 07 07:13:27.077608 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:27.077586    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c66c8b568-dwbks_3de2d723-cc06-48f1-a5cb-28994950a44b/copy-manifests/0.log"
Jul 07 07:13:27.170390 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:27.170354    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-b6stw_0d222fa8-990f-4c76-b79a-abbca96a5cdd/postgres/0.log"
Jul 07 07:13:28.687812 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:28.687735    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-856bfb9fcf-zdcjc_d5ed9d72-fcb9-43a4-9cad-9a5f217b0437/manager/0.log"
Jul 07 07:13:29.012108 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:29.012015    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74994db979-jwcx7_ce07d74d-6b0a-4e94-b31c-c9553c5f3a1b/kube-auth-proxy/0.log"
Jul 07 07:13:29.099091 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:29.099051    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-rklkm_be371c12-0c30-4424-8f5d-ac6a5883f509/payload-pre-processing/0.log"
Jul 07 07:13:29.164398 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:29.164367    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-lk6nm_28572129-d52e-4589-9993-2adf95c458e2/payload-processing/0.log"
Jul 07 07:13:30.530729 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:30.530701    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74994db979-jwcx7_ce07d74d-6b0a-4e94-b31c-c9553c5f3a1b/kube-auth-proxy/0.log"
Jul 07 07:13:30.632174 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:30.632087    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-rklkm_be371c12-0c30-4424-8f5d-ac6a5883f509/payload-pre-processing/0.log"
Jul 07 07:13:30.698730 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:30.698698    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-lk6nm_28572129-d52e-4589-9993-2adf95c458e2/payload-processing/0.log"
Jul 07 07:13:35.345891 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:35.345858    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-54wzm_2060cba3-4269-4ecd-a03c-7a51fad57d0c/kube-multus/0.log"
Jul 07 07:13:35.368194 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:35.368170    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b6cs5_0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c/kube-multus-additional-cni-plugins/0.log"
Jul 07 07:13:35.387238 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:35.387216    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b6cs5_0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c/egress-router-binary-copy/0.log"
Jul 07 07:13:35.406559 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:35.406526    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b6cs5_0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c/cni-plugins/0.log"
Jul 07 07:13:35.426857 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:35.426839    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b6cs5_0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c/bond-cni-plugin/0.log"
Jul 07 07:13:35.451211 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:35.451187    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b6cs5_0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c/routeoverride-cni/0.log"
Jul 07 07:13:35.477133 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:35.477097    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b6cs5_0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c/whereabouts-cni-bincopy/0.log"
Jul 07 07:13:35.505408 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:35.505389    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b6cs5_0be65e1c-ab3b-4ad1-9c38-5b96dadf0f6c/whereabouts-cni/0.log"
Jul 07 07:13:35.896843 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:35.896816    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mx8qb_7c1e103c-f156-4b8a-8426-f18974595c51/network-metrics-daemon/0.log"
Jul 07 07:13:35.911324 ip-10-0-133-92 kubenswrapper[2563]: I0707 07:13:35.911297    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mx8qb_7c1e103c-f156-4b8a-8426-f18974595c51/kube-rbac-proxy/0.log"