Jul 14 19:47:25.500022 ip-10-0-133-168 systemd[1]: Starting Kubernetes Kubelet...
Jul 14 19:47:25.897233 ip-10-0-133-168 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 19:47:25.897233 ip-10-0-133-168 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 14 19:47:25.897233 ip-10-0-133-168 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 19:47:25.897233 ip-10-0-133-168 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 14 19:47:25.897233 ip-10-0-133-168 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 19:47:25.898191 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.898106    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 14 19:47:25.902245 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902229    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 19:47:25.902245 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902245    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 19:47:25.902306 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902250    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 19:47:25.902306 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902252    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 19:47:25.902306 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902256    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 19:47:25.902306 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902258    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 19:47:25.902306 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902261    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 19:47:25.902306 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902264    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 19:47:25.902306 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902266    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 19:47:25.902306 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902269    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 19:47:25.902306 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902271    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 19:47:25.902306 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902274    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 19:47:25.902306 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902277    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 19:47:25.902306 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902281    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 19:47:25.902306 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902285    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 19:47:25.902306 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902288    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 19:47:25.902306 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902290    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 19:47:25.902306 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902293    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 19:47:25.902306 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902297    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 19:47:25.902306 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902299    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 19:47:25.902306 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902302    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 19:47:25.902774 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902305    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 19:47:25.902774 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902308    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 19:47:25.902774 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902311    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 19:47:25.902774 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902314    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 19:47:25.902774 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902316    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 19:47:25.902774 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902319    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 19:47:25.902774 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902322    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 19:47:25.902774 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902325    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 19:47:25.902774 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902327    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 19:47:25.902774 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902330    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 19:47:25.902774 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902334    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 19:47:25.902774 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902338    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 19:47:25.902774 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902340    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 19:47:25.902774 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902343    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 19:47:25.902774 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902347    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 19:47:25.902774 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902350    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 19:47:25.902774 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902353    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 19:47:25.902774 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902355    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 19:47:25.902774 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902358    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 19:47:25.903253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902360    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 19:47:25.903253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902364    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 19:47:25.903253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902367    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 19:47:25.903253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902370    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 19:47:25.903253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902373    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 19:47:25.903253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902376    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 19:47:25.903253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902378    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 19:47:25.903253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902380    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 19:47:25.903253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902383    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 19:47:25.903253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902385    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 19:47:25.903253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902388    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 19:47:25.903253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902390    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 19:47:25.903253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902393    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 19:47:25.903253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902395    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 19:47:25.903253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902399    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 19:47:25.903253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902401    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 19:47:25.903253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902404    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 19:47:25.903253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902406    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 19:47:25.903253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902409    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 19:47:25.903253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902412    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 19:47:25.903760 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902414    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 19:47:25.903760 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902416    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 19:47:25.903760 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902419    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 19:47:25.903760 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902421    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 19:47:25.903760 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902425    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 19:47:25.903760 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902427    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 19:47:25.903760 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902430    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 19:47:25.903760 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902432    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 19:47:25.903760 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902434    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 19:47:25.903760 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902436    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 19:47:25.903760 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902439    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 19:47:25.903760 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902441    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 19:47:25.903760 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902444    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 19:47:25.903760 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902446    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 19:47:25.903760 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902449    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 19:47:25.903760 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902451    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 19:47:25.903760 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902454    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 19:47:25.903760 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902456    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 19:47:25.903760 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902458    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 19:47:25.903760 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902461    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 19:47:25.904249 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902463    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 19:47:25.904249 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902465    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 19:47:25.904249 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902468    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 19:47:25.904249 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902470    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 19:47:25.904249 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902473    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 19:47:25.904249 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902476    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 19:47:25.904249 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902479    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 19:47:25.904249 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902482    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 19:47:25.904249 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902484    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 19:47:25.904249 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902487    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 19:47:25.904249 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902489    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 19:47:25.904249 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902491    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 19:47:25.904249 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902494    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 19:47:25.904249 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902496    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 19:47:25.904249 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902498    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 19:47:25.904249 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902501    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 19:47:25.904249 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902504    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 19:47:25.904249 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902506    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 19:47:25.904249 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.902509    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 19:47:25.904765 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904281    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 19:47:25.904765 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904289    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 19:47:25.904765 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904292    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 19:47:25.904765 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904296    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 19:47:25.904765 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904299    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 19:47:25.904765 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904302    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 19:47:25.904765 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904305    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 19:47:25.904765 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904307    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 19:47:25.904765 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904309    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 19:47:25.904765 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904312    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 19:47:25.904765 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904314    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 19:47:25.904765 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904317    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 19:47:25.904765 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904319    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 19:47:25.904765 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904321    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 19:47:25.904765 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904324    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 19:47:25.904765 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904328    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 19:47:25.904765 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904331    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 19:47:25.904765 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904334    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 19:47:25.904765 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904338    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 19:47:25.904765 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904341    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 19:47:25.905714 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904344    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 19:47:25.905714 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904346    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 19:47:25.905714 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904349    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 19:47:25.905714 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904351    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 19:47:25.905714 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904353    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 19:47:25.905714 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904356    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 19:47:25.905714 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904358    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 19:47:25.905714 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904360    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 19:47:25.905714 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904363    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 19:47:25.905714 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904365    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 19:47:25.905714 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904367    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 19:47:25.905714 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904370    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 19:47:25.905714 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904373    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 19:47:25.905714 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904376    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 19:47:25.905714 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904379    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 19:47:25.905714 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904382    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 19:47:25.905714 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904384    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 19:47:25.905714 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904387    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 19:47:25.905714 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904390    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 19:47:25.905714 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904392    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 19:47:25.906302 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904394    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 19:47:25.906302 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904397    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 19:47:25.906302 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904399    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 19:47:25.906302 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904402    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 19:47:25.906302 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904405    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 19:47:25.906302 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904407    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 19:47:25.906302 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904410    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 19:47:25.906302 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904414    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 19:47:25.906302 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904417    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 19:47:25.906302 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904420    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 19:47:25.906302 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904423    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 19:47:25.906302 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904425    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 19:47:25.906302 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904428    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 19:47:25.906302 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904430    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 19:47:25.906302 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904433    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 19:47:25.906302 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904435    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 19:47:25.906302 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904438    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 19:47:25.906302 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904440    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 19:47:25.906302 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904442    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 19:47:25.906302 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904445    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 19:47:25.906827 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904447    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 19:47:25.906827 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904450    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 19:47:25.906827 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904452    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 19:47:25.906827 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904454    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 19:47:25.906827 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904457    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 19:47:25.906827 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904461    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 19:47:25.906827 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904463    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 19:47:25.906827 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904466    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 19:47:25.906827 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904468    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 19:47:25.906827 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904471    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 19:47:25.906827 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904473    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 19:47:25.906827 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904475    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 19:47:25.906827 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904478    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 19:47:25.906827 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904480    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 19:47:25.906827 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904484    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 19:47:25.906827 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904488    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 19:47:25.906827 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904507    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 19:47:25.906827 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904512    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 19:47:25.906827 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904515    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 19:47:25.906827 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904517    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 19:47:25.907310 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904520    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 19:47:25.907310 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904523    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 19:47:25.907310 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904526    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 19:47:25.907310 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904529    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 19:47:25.907310 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904532    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 19:47:25.907310 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904535    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 19:47:25.907310 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904538    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 19:47:25.907310 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904540    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 19:47:25.907310 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904543    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 19:47:25.907310 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904545    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 19:47:25.907310 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904547    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 19:47:25.907310 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904550    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 19:47:25.907310 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904554    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 19:47:25.907310 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904557    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 19:47:25.907310 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904560    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 19:47:25.907310 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904563    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 19:47:25.907310 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904566    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 19:47:25.907310 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904569    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 19:47:25.907310 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.904572    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 19:47:25.907792 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906177    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 19:47:25.907792 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906187    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 19:47:25.907792 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906191    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 19:47:25.907792 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906194    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 19:47:25.907792 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906197    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 19:47:25.907792 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906199    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 19:47:25.907792 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906202    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 19:47:25.907792 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906205    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 19:47:25.907792 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906207    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 19:47:25.907792 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906210    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 19:47:25.907792 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906212    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 19:47:25.907792 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906215    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 19:47:25.907792 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906217    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 19:47:25.907792 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906220    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 19:47:25.907792 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906222    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 19:47:25.907792 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906225    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 19:47:25.907792 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906227    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 19:47:25.907792 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906230    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 19:47:25.907792 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906232    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 19:47:25.907792 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906235    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 19:47:25.908286 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906237    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 19:47:25.908286 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906240    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 19:47:25.908286 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906242    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 19:47:25.908286 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906245    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 19:47:25.908286 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906247    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 19:47:25.908286 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906251    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 19:47:25.908286 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906254    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 19:47:25.908286 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906258    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 19:47:25.908286 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906260    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 19:47:25.908286 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906263    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 19:47:25.908286 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906266    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 19:47:25.908286 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906269    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 19:47:25.908286 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906272    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 19:47:25.908286 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906275    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 19:47:25.908286 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906278    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 19:47:25.908286 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906281    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 19:47:25.908286 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906284    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 19:47:25.908286 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906287    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 19:47:25.908286 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906289    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 19:47:25.908286 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906292    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 19:47:25.908793 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906295    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 19:47:25.908793 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906297    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 19:47:25.908793 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906300    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 19:47:25.908793 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906303    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 19:47:25.908793 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906306    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 19:47:25.908793 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906310    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 19:47:25.908793 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906312    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 19:47:25.908793 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906315    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 19:47:25.908793 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906318    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 19:47:25.908793 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906321    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 19:47:25.908793 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906323    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 19:47:25.908793 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906326    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 19:47:25.908793 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906328    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 19:47:25.908793 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906331    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 19:47:25.908793 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906334    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 19:47:25.908793 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906336    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 19:47:25.908793 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906339    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 19:47:25.908793 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906341    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 19:47:25.908793 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906345    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 19:47:25.909253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906348    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 19:47:25.909253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906351    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 19:47:25.909253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906353    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 19:47:25.909253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906356    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 19:47:25.909253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906359    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 19:47:25.909253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906361    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 19:47:25.909253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906365    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 19:47:25.909253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906367    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 19:47:25.909253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906370    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 19:47:25.909253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906372    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 19:47:25.909253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906375    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 19:47:25.909253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906378    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 19:47:25.909253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906380    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 19:47:25.909253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906383    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 19:47:25.909253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906386    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 19:47:25.909253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906389    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 19:47:25.909253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906392    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 19:47:25.909253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906396    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 19:47:25.909253 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906398    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 19:47:25.909729 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906401    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 19:47:25.909729 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906404    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 19:47:25.909729 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906406    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 19:47:25.909729 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906408    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 19:47:25.909729 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906412    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 19:47:25.909729 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906414    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 19:47:25.909729 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906417    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 19:47:25.909729 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906419    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 19:47:25.909729 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906421    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 19:47:25.909729 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906424    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 19:47:25.909729 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906426    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 19:47:25.909729 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906429    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 19:47:25.909729 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906432    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 19:47:25.909729 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906434    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 19:47:25.909729 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906436    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 19:47:25.909729 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906438    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 19:47:25.909729 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906441    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 19:47:25.909729 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906443    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 19:47:25.909729 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906446    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 19:47:25.909729 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906448    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 19:47:25.910210 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.906451    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 19:47:25.910210 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906530    2560 flags.go:64] FLAG: --address="0.0.0.0"
Jul 14 19:47:25.910210 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906537    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 14 19:47:25.910210 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906544    2560 flags.go:64] FLAG: --anonymous-auth="true"
Jul 14 19:47:25.910210 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906548    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 14 19:47:25.910210 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906552    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 14 19:47:25.910210 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906555    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 14 19:47:25.910210 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906561    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 14 19:47:25.910210 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906568    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 14 19:47:25.910210 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906572    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 14 19:47:25.910210 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906575    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 14 19:47:25.910210 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906578    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 14 19:47:25.910210 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906581    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 14 19:47:25.910210 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906585    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 14 19:47:25.910210 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906587    2560 flags.go:64] FLAG: --cgroup-root=""
Jul 14 19:47:25.910210 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906590    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 14 19:47:25.910210 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906593    2560 flags.go:64] FLAG: --client-ca-file=""
Jul 14 19:47:25.910210 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906599    2560 flags.go:64] FLAG: --cloud-provider="external"
Jul 14 19:47:25.910210 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906602    2560 flags.go:64] FLAG: --cluster-dns="[]"
Jul 14 19:47:25.910210 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906625    2560 flags.go:64] FLAG: --cluster-domain=""
Jul 14 19:47:25.910210 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906628    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 14 19:47:25.910210 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906631    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 14 19:47:25.910210 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906634    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 14 19:47:25.910210 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906637    2560 flags.go:64] FLAG: --container-log-max-files="5"
Jul 14 19:47:25.910799 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906641    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 14 19:47:25.910799 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906644    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 14 19:47:25.910799 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906647    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 14 19:47:25.910799 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906651    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 14 19:47:25.910799 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906653    2560 flags.go:64] FLAG: --contention-profiling="false"
Jul 14 19:47:25.910799 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906656    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 14 19:47:25.910799 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906659    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 14 19:47:25.910799 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906662    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 14 19:47:25.910799 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906665    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 14 19:47:25.910799 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906669    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 14 19:47:25.910799 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906672    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 14 19:47:25.910799 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906675    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 14 19:47:25.910799 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906678    2560 flags.go:64] FLAG: --enable-load-reader="false"
Jul 14 19:47:25.910799 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906680    2560 flags.go:64] FLAG: --enable-server="true"
Jul 14 19:47:25.910799 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906683    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 14 19:47:25.910799 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906688    2560 flags.go:64] FLAG: --event-burst="100"
Jul 14 19:47:25.910799 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906691    2560 flags.go:64] FLAG: --event-qps="50"
Jul 14 19:47:25.910799 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906694    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 14 19:47:25.910799 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906719    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 14 19:47:25.910799 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906722    2560 flags.go:64] FLAG: --eviction-hard=""
Jul 14 19:47:25.910799 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906726    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 14 19:47:25.910799 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906729    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 14 19:47:25.910799 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906732    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 14 19:47:25.910799 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906735    2560 flags.go:64] FLAG: --eviction-soft=""
Jul 14 19:47:25.910799 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906738    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 14 19:47:25.911394 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906742    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 14 19:47:25.911394 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906746    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 14 19:47:25.911394 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906749    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 14 19:47:25.911394 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906751    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 14 19:47:25.911394 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906754    2560 flags.go:64] FLAG: --fail-swap-on="true"
Jul 14 19:47:25.911394 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906757    2560 flags.go:64] FLAG: --feature-gates=""
Jul 14 19:47:25.911394 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906760    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 14 19:47:25.911394 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906763    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 14 19:47:25.911394 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906767    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 14 19:47:25.911394 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906769    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 14 19:47:25.911394 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906772    2560 flags.go:64] FLAG: --healthz-port="10248"
Jul 14 19:47:25.911394 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906775    2560 flags.go:64] FLAG: --help="false"
Jul 14 19:47:25.911394 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906778    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-133-168.ec2.internal"
Jul 14 19:47:25.911394 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906781    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 14 19:47:25.911394 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906784    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 14 19:47:25.911394 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906787    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 14 19:47:25.911394 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906790    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 14 19:47:25.911394 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906793    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 14 19:47:25.911394 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906796    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 14 19:47:25.911394 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906799    2560 flags.go:64] FLAG: --image-service-endpoint=""
Jul 14 19:47:25.911394 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906802    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 14 19:47:25.911394 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906804    2560 flags.go:64] FLAG: --kube-api-burst="100"
Jul 14 19:47:25.911394 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906807    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 14 19:47:25.912031 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906810    2560 flags.go:64] FLAG: --kube-api-qps="50"
Jul 14 19:47:25.912031 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906812    2560 flags.go:64] FLAG: --kube-reserved=""
Jul 14 19:47:25.912031 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906815    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 14 19:47:25.912031 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906818    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 14 19:47:25.912031 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906821    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 14 19:47:25.912031 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906824    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 14 19:47:25.912031 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906826    2560 flags.go:64] FLAG: --lock-file=""
Jul 14 19:47:25.912031 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906829    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 14 19:47:25.912031 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906832    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 14 19:47:25.912031 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906837    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 14 19:47:25.912031 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906842    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 14 19:47:25.912031 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906846    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 14 19:47:25.912031 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906848    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 14 19:47:25.912031 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906851    2560 flags.go:64] FLAG: --logging-format="text"
Jul 14 19:47:25.912031 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906854    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 14 19:47:25.912031 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906857    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 14 19:47:25.912031 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906860    2560 flags.go:64] FLAG: --manifest-url=""
Jul 14 19:47:25.912031 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906863    2560 flags.go:64] FLAG: --manifest-url-header=""
Jul 14 19:47:25.912031 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906867    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 14 19:47:25.912031 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906870    2560 flags.go:64] FLAG: --max-open-files="1000000"
Jul 14 19:47:25.912031 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906874    2560 flags.go:64] FLAG: --max-pods="110"
Jul 14 19:47:25.912031 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906877    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 14 19:47:25.912031 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906881    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 14 19:47:25.912031 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906884    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 14 19:47:25.912031 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906887    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 14 19:47:25.912653 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906890    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 14 19:47:25.912653 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906893    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 14 19:47:25.912653 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906895    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 14 19:47:25.912653 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906901    2560 flags.go:64] FLAG: --node-status-max-images="50"
Jul 14 19:47:25.912653 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906904    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 14 19:47:25.912653 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906907    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 14 19:47:25.912653 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906910    2560 flags.go:64] FLAG: --pod-cidr=""
Jul 14 19:47:25.912653 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906912    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 14 19:47:25.912653 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906918    2560 flags.go:64] FLAG: --pod-manifest-path=""
Jul 14 19:47:25.912653 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906921    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 14 19:47:25.912653 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906924    2560 flags.go:64] FLAG: --pods-per-core="0"
Jul 14 19:47:25.912653 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906926    2560 flags.go:64] FLAG: --port="10250"
Jul 14 19:47:25.912653 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906929    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 14 19:47:25.912653 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906932    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ccef7271a323e373"
Jul 14 19:47:25.912653 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906935    2560 flags.go:64] FLAG: --qos-reserved=""
Jul 14 19:47:25.912653 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906938    2560 flags.go:64] FLAG: --read-only-port="10255"
Jul 14 19:47:25.912653 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906941    2560 flags.go:64] FLAG: --register-node="true"
Jul 14 19:47:25.912653 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906945    2560 flags.go:64] FLAG: --register-with-taints=""
Jul 14 19:47:25.912653 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906949    2560 flags.go:64] FLAG: --registry-burst="10"
Jul 14 19:47:25.912653 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906952    2560 flags.go:64] FLAG: --registry-qps="5"
Jul 14 19:47:25.912653 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906955    2560 flags.go:64] FLAG: --reserved-cpus=""
Jul 14 19:47:25.912653 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906958    2560 flags.go:64] FLAG: --reserved-memory=""
Jul 14 19:47:25.912653 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906961    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 14 19:47:25.912653 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906964    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 14 19:47:25.912653 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906967    2560 flags.go:64] FLAG: --rotate-certificates="false"
Jul 14 19:47:25.913301 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906970    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 14 19:47:25.913301 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906972    2560 flags.go:64] FLAG: --runonce="false"
Jul 14 19:47:25.913301 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906975    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 14 19:47:25.913301 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906978    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 14 19:47:25.913301 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906982    2560 flags.go:64] FLAG: --seccomp-default="false"
Jul 14 19:47:25.913301 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906985    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 14 19:47:25.913301 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906987    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 14 19:47:25.913301 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906990    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 14 19:47:25.913301 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906993    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 14 19:47:25.913301 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906996    2560 flags.go:64] FLAG: --storage-driver-password="root"
Jul 14 19:47:25.913301 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.906999    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 14 19:47:25.913301 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.907002    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 14 19:47:25.913301 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.907004    2560 flags.go:64] FLAG: --storage-driver-user="root"
Jul 14 19:47:25.913301 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.907008    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 14 19:47:25.913301 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.907011    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 14 19:47:25.913301 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.907014    2560 flags.go:64] FLAG: --system-cgroups=""
Jul 14 19:47:25.913301 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.907016    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 14 19:47:25.913301 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.907022    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 14 19:47:25.913301 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.907024    2560 flags.go:64] FLAG: --tls-cert-file=""
Jul 14 19:47:25.913301 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.907027    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 14 19:47:25.913301 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.907031    2560 flags.go:64] FLAG: --tls-min-version=""
Jul 14 19:47:25.913301 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.907033    2560 flags.go:64] FLAG: --tls-private-key-file=""
Jul 14 19:47:25.913301 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.907036    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 14 19:47:25.913301 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.907039    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 14 19:47:25.913301 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.907043    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 14 19:47:25.914375 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.907046    2560 flags.go:64] FLAG: --v="2"
Jul 14 19:47:25.914375 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.907049    2560 flags.go:64] FLAG: --version="false"
Jul 14 19:47:25.914375 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.907054    2560 flags.go:64] FLAG: --vmodule=""
Jul 14 19:47:25.914375 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.907058    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 14 19:47:25.914375 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.907061    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 14 19:47:25.914375 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907171    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 19:47:25.914375 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907175    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 19:47:25.914375 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907178    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 19:47:25.914375 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907181    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 19:47:25.914375 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907183    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 19:47:25.914375 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907185    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 19:47:25.914375 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907190    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 19:47:25.914375 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907193    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 19:47:25.914375 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907195    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 19:47:25.914375 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907198    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 19:47:25.914375 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907201    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 19:47:25.914375 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907203    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 19:47:25.914375 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907206    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 19:47:25.914375 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907208    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 19:47:25.914375 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907211    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 19:47:25.914375 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907213    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 19:47:25.915290 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907216    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 19:47:25.915290 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907218    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 19:47:25.915290 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907221    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 19:47:25.915290 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907224    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 19:47:25.915290 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907228    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 19:47:25.915290 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907231    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 19:47:25.915290 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907234    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 19:47:25.915290 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907236    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 19:47:25.915290 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907239    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 19:47:25.915290 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907242    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 19:47:25.915290 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907246    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 19:47:25.915290 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907250    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 19:47:25.915290 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907252    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 19:47:25.915290 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907255    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 19:47:25.915290 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907258    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 19:47:25.915290 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907261    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 19:47:25.915290 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907263    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 19:47:25.915290 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907266    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 19:47:25.915290 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907268    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 19:47:25.916163 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907271    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 19:47:25.916163 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907273    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 19:47:25.916163 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907276    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 19:47:25.916163 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907279    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 19:47:25.916163 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907282    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 19:47:25.916163 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907284    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 19:47:25.916163 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907287    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 19:47:25.916163 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907289    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 19:47:25.916163 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907292    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 19:47:25.916163 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907294    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 19:47:25.916163 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907296    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 19:47:25.916163 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907299    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 19:47:25.916163 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907301    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 19:47:25.916163 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907304    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 19:47:25.916163 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907306    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 19:47:25.916163 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907308    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 19:47:25.916163 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907311    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 19:47:25.916163 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907314    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 19:47:25.916163 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907316    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 19:47:25.916163 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907319    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 19:47:25.917086 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907321    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 19:47:25.917086 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907324    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 19:47:25.917086 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907327    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 19:47:25.917086 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907331    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 19:47:25.917086 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907334    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 19:47:25.917086 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907336    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 19:47:25.917086 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907339    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 19:47:25.917086 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907341    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 19:47:25.917086 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907344    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 19:47:25.917086 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907347    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 19:47:25.917086 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907349    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 19:47:25.917086 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907351    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 19:47:25.917086 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907354    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 19:47:25.917086 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907357    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 19:47:25.917086 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907359    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 19:47:25.917086 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907364    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 19:47:25.917086 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907367    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 19:47:25.917086 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907369    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 19:47:25.917086 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907371    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 19:47:25.917086 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907374    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 19:47:25.917597 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907377    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 19:47:25.917597 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907381    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 19:47:25.917597 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907384    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 19:47:25.917597 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907388    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 19:47:25.917597 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907391    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 19:47:25.917597 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907394    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 19:47:25.917597 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907397    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 19:47:25.917597 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907399    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 19:47:25.917597 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907402    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 19:47:25.917597 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907405    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 19:47:25.917597 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907407    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 19:47:25.917597 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907410    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 19:47:25.917597 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907413    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 19:47:25.917597 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907416    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 19:47:25.917597 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907418    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 19:47:25.917597 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907422    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 19:47:25.917597 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907425    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 19:47:25.917597 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907428    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 19:47:25.917597 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907431    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 19:47:25.918081 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907434    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 19:47:25.918081 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907436    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 19:47:25.918081 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907439    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 19:47:25.918081 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907442    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 19:47:25.918081 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.907445    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 19:47:25.918081 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.908214    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 19:47:25.918081 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.915525    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 14 19:47:25.918081 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.915545    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 14 19:47:25.918081 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.915572    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 14 19:47:25.918081 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.915581    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 14 19:47:25.918081 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915666    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 19:47:25.918081 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915676    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 19:47:25.918081 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915682    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 19:47:25.918081 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915687    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 19:47:25.918564 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915692    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 19:47:25.918564 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915719    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 19:47:25.918564 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915724    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 19:47:25.918564 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915728    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 19:47:25.918564 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915732    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 19:47:25.918564 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915735    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 19:47:25.918564 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915739    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 19:47:25.918564 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915742    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 19:47:25.918564 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915746    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 19:47:25.918564 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915750    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 19:47:25.918564 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915754    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 19:47:25.918564 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915758    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 19:47:25.918564 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915763    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 19:47:25.918564 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915767    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 19:47:25.918564 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915771    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 19:47:25.918564 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915775    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 19:47:25.918564 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915779    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 19:47:25.918564 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915784    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 19:47:25.918564 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915788    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 19:47:25.918564 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915792    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 19:47:25.919404 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915796    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 19:47:25.919404 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915800    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 19:47:25.919404 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915804    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 19:47:25.919404 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915808    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 19:47:25.919404 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915812    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 19:47:25.919404 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915819    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 19:47:25.919404 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915825    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 19:47:25.919404 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915831    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 19:47:25.919404 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915835    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 19:47:25.919404 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915840    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 19:47:25.919404 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915844    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 19:47:25.919404 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915849    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 19:47:25.919404 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915853    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 19:47:25.919404 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915857    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 19:47:25.919404 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915862    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 19:47:25.919404 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915866    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 19:47:25.919404 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915870    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 19:47:25.919404 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915874    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 19:47:25.919404 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915878    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 19:47:25.920125 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915882    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 19:47:25.920125 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915886    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 19:47:25.920125 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915890    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 19:47:25.920125 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915894    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 19:47:25.920125 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915898    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 19:47:25.920125 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915903    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 19:47:25.920125 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915907    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 19:47:25.920125 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915911    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 19:47:25.920125 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915915    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 19:47:25.920125 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915919    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 19:47:25.920125 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915923    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 19:47:25.920125 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915926    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 19:47:25.920125 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915931    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 19:47:25.920125 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915935    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 19:47:25.920125 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915939    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 19:47:25.920125 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915943    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 19:47:25.920125 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915947    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 19:47:25.920125 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915951    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 19:47:25.920125 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915956    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 19:47:25.920125 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915961    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 19:47:25.920630 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915965    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 19:47:25.920630 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915969    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 19:47:25.920630 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915973    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 19:47:25.920630 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915977    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 19:47:25.920630 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915982    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 19:47:25.920630 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915986    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 19:47:25.920630 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915990    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 19:47:25.920630 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.915996    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 19:47:25.920630 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916002    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 19:47:25.920630 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916006    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 19:47:25.920630 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916011    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 19:47:25.920630 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916016    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 19:47:25.920630 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916020    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 19:47:25.920630 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916024    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 19:47:25.920630 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916028    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 19:47:25.920630 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916032    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 19:47:25.920630 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916036    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 19:47:25.920630 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916040    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 19:47:25.920630 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916044    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 19:47:25.920630 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916048    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 19:47:25.921184 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916052    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 19:47:25.921184 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916056    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 19:47:25.921184 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916060    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 19:47:25.921184 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916064    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 19:47:25.921184 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916068    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 19:47:25.921184 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916072    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 19:47:25.921184 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916076    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 19:47:25.921184 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916080    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 19:47:25.921184 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916084    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 19:47:25.921184 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916088    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 19:47:25.921184 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916093    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 19:47:25.921184 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916098    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 19:47:25.921184 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916103    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 19:47:25.921184 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916107    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 19:47:25.921184 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916112    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 19:47:25.921184 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916116    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 19:47:25.921574 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.916124    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 19:47:25.921574 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916323    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 19:47:25.921574 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916333    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 19:47:25.921574 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916337    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 19:47:25.921574 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916342    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 19:47:25.921574 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916347    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 19:47:25.921574 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916352    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 19:47:25.921574 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916356    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 19:47:25.921574 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916361    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 19:47:25.921574 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916365    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 19:47:25.921574 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916370    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 19:47:25.921574 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916374    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 19:47:25.921574 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916378    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 19:47:25.921574 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916382    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 19:47:25.921574 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916386    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 19:47:25.921957 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916390    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 19:47:25.921957 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916394    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 19:47:25.921957 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916399    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 19:47:25.921957 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916403    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 19:47:25.921957 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916406    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 19:47:25.921957 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916411    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 19:47:25.921957 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916415    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 19:47:25.921957 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916419    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 19:47:25.921957 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916423    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 19:47:25.921957 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916426    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 19:47:25.921957 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916430    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 19:47:25.921957 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916434    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 19:47:25.921957 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916439    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 19:47:25.921957 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916443    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 19:47:25.921957 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916448    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 19:47:25.921957 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916453    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 19:47:25.921957 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916457    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 19:47:25.921957 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916461    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 19:47:25.921957 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916465    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 19:47:25.921957 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916469    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 19:47:25.922447 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916474    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 19:47:25.922447 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916478    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 19:47:25.922447 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916483    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 19:47:25.922447 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916487    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 19:47:25.922447 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916491    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 19:47:25.922447 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916503    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 19:47:25.922447 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916508    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 19:47:25.922447 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916513    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 19:47:25.922447 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916517    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 19:47:25.922447 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916521    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 19:47:25.922447 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916525    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 19:47:25.922447 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916529    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 19:47:25.922447 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916534    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 19:47:25.922447 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916537    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 19:47:25.922447 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916542    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 19:47:25.922447 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916546    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 19:47:25.922447 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916550    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 19:47:25.922447 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916554    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 19:47:25.922447 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916559    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 19:47:25.922447 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916562    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 19:47:25.922964 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916566    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 19:47:25.922964 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916570    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 19:47:25.922964 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916574    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 19:47:25.922964 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916579    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 19:47:25.922964 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916583    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 19:47:25.922964 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916587    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 19:47:25.922964 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916593    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 19:47:25.922964 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916600    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 19:47:25.922964 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916604    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 19:47:25.922964 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916607    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 19:47:25.922964 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916611    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 19:47:25.922964 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916616    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 19:47:25.922964 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916620    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 19:47:25.922964 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916624    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 19:47:25.922964 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916628    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 19:47:25.922964 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916632    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 19:47:25.922964 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916636    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 19:47:25.922964 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916642    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 19:47:25.922964 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916647    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 19:47:25.922964 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916651    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 19:47:25.923441 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916657    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 19:47:25.923441 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916663    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 19:47:25.923441 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916668    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 19:47:25.923441 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916672    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 19:47:25.923441 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916676    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 19:47:25.923441 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916680    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 19:47:25.923441 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916684    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 19:47:25.923441 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916688    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 19:47:25.923441 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916692    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 19:47:25.923441 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916714    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 19:47:25.923441 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916720    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 19:47:25.923441 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916725    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 19:47:25.923441 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916730    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 19:47:25.923441 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916734    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 19:47:25.923441 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916739    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 19:47:25.923441 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916744    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 19:47:25.923441 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916749    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 19:47:25.923441 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916753    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 19:47:25.923441 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916757    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 19:47:25.923947 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916761    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 19:47:25.923947 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916767    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 19:47:25.923947 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916771    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 19:47:25.923947 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916775    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 19:47:25.923947 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916780    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 19:47:25.923947 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:25.916784    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 19:47:25.923947 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.916792    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 19:47:25.923947 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.916946    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 14 19:47:25.923947 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.920373    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 14 19:47:25.923947 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.921837    2560 server.go:1019] "Starting client certificate rotation"
Jul 14 19:47:25.923947 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.921939    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 14 19:47:25.923947 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.921992    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 14 19:47:25.924239 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.924211    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 14 19:47:25.927248 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.927233    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 14 19:47:25.940308 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.940287    2560 log.go:25] "Validated CRI v1 runtime API"
Jul 14 19:47:25.942315 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.942299    2560 log.go:25] "Validated CRI v1 image API"
Jul 14 19:47:25.943080 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.943068    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 14 19:47:25.945530 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.945512    2560 fs.go:135] Filesystem UUIDs: map[31c681d5-54e7-4f8f-a7af-452d7f7bc764:/dev/nvme0n1p3 51c026fa-c91b-4497-9288-02081e3fa849:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 14 19:47:25.945585 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.945531    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 14 19:47:25.949232 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.949216    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 14 19:47:25.951100 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.950990    2560 manager.go:216] Machine: {Timestamp:2026-07-14 19:47:25.949074492 +0000 UTC m=+0.336849782 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3081132 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2af25f12047a60ec82b7b371fbb287 SystemUUID:ec2af25f-1204-7a60-ec82-b7b371fbb287 BootID:e4fe46a8-8831-4088-89b6-906feef696fb Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1a:db:89:13:6b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1a:db:89:13:6b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:79:ca:f2:1b:4b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 14 19:47:25.951100 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.951098    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 14 19:47:25.951209 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.951174    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 14 19:47:25.952156 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.952131    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 14 19:47:25.952292 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.952159    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-168.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 14 19:47:25.953452 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.953441    2560 topology_manager.go:138] "Creating topology manager with none policy"
Jul 14 19:47:25.953492 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.953454    2560 container_manager_linux.go:309] "Creating device plugin manager"
Jul 14 19:47:25.953492 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.953461    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 14 19:47:25.954195 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.954184    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 14 19:47:25.955490 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.955481    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 14 19:47:25.958947 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.958936    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 14 19:47:25.959071 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.959061    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 14 19:47:25.959105 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.959094    2560 kubelet.go:486] "Attempting to sync node with API server"
Jul 14 19:47:25.959139 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.959112    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 14 19:47:25.959139 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.959126    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 14 19:47:25.959139 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.959136    2560 kubelet.go:398] "Adding apiserver pod source"
Jul 14 19:47:25.959238 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.959148    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 14 19:47:25.959528 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.959516    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 14 19:47:25.959568 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.959538    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 14 19:47:25.962132 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.962116    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 14 19:47:25.963790 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.963777    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 14 19:47:25.963829 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.963807    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 14 19:47:25.964430 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.964419    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 14 19:47:25.964461 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.964437    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 14 19:47:25.964461 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.964444    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 14 19:47:25.964461 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.964448    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 14 19:47:25.964461 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.964456    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 14 19:47:25.964461 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.964462    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 14 19:47:25.964870 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.964860    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 14 19:47:25.964906 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.964870    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 14 19:47:25.964906 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.964877    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 14 19:47:25.964906 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.964883    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 14 19:47:25.964906 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.964894    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 14 19:47:25.964906 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.964901    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 14 19:47:25.965553 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.965541    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 14 19:47:25.965583 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.965555    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 14 19:47:25.968786 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.968772    2560 server.go:1268] "Started kubelet"
Jul 14 19:47:25.968916 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.968891    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 14 19:47:25.969383 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.968999    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 14 19:47:25.969383 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.969063    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 14 19:47:25.969823 ip-10-0-133-168 systemd[1]: Started Kubernetes Kubelet.
Jul 14 19:47:25.970432 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.970377    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-168.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 14 19:47:25.970960 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.970944    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 14 19:47:25.971020 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:25.970987    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 14 19:47:25.971020 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:25.970994    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-168.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 14 19:47:25.971363 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.971350    2560 server.go:310] "Adding debug handlers to kubelet server"
Jul 14 19:47:25.976182 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:25.976155    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 14 19:47:25.976580 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.976511    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 14 19:47:25.976691 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.976593    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 14 19:47:25.976783 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:25.976727    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-168.ec2.internal\" not found"
Jul 14 19:47:25.977342 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.977320    2560 factory.go:55] Registering systemd factory
Jul 14 19:47:25.977342 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.977338    2560 factory.go:223] Registration of the systemd container factory successfully
Jul 14 19:47:25.977487 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.977474    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 14 19:47:25.977542 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.977477    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 14 19:47:25.977542 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.977502    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 14 19:47:25.977542 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.977510    2560 factory.go:153] Registering CRI-O factory
Jul 14 19:47:25.977542 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.977521    2560 factory.go:223] Registration of the crio container factory successfully
Jul 14 19:47:25.977756 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.977566    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 14 19:47:25.977756 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.977591    2560 factory.go:103] Registering Raw factory
Jul 14 19:47:25.977756 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.977592    2560 reconstruct.go:97] "Volume reconstruction finished"
Jul 14 19:47:25.977756 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.977603    2560 manager.go:1196] Started watching for new ooms in manager
Jul 14 19:47:25.977756 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.977603    2560 reconciler.go:29] "Reconciler: start to sync state"
Jul 14 19:47:25.978026 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:25.977156    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-168.ec2.internal.18c23fb44a80a540  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-168.ec2.internal,UID:ip-10-0-133-168.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-168.ec2.internal,},FirstTimestamp:2026-07-14 19:47:25.968745792 +0000 UTC m=+0.356521081,LastTimestamp:2026-07-14 19:47:25.968745792 +0000 UTC m=+0.356521081,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-168.ec2.internal,}"
Jul 14 19:47:25.978219 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.978206    2560 manager.go:349] Starting recovery of all containers
Jul 14 19:47:25.979689 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.979656    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4fltr"
Jul 14 19:47:25.986196 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:25.986040    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 14 19:47:25.986300 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:25.986192    2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-168.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 14 19:47:25.987149 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.987127    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4fltr"
Jul 14 19:47:25.987567 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.987545    2560 manager.go:354] Recovery completed
Jul 14 19:47:25.992165 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.992150    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 19:47:25.994143 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.994126    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 19:47:25.994215 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.994159    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 19:47:25.994215 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.994182    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasSufficientPID"
Jul 14 19:47:25.994679 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.994665    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 14 19:47:25.994760 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.994679    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 14 19:47:25.994760 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.994715    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 14 19:47:25.994760 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.994723    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 19:47:25.995629 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.995615    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 19:47:25.995690 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.995638    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 19:47:25.995690 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.995646    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasSufficientPID"
Jul 14 19:47:25.995987 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:25.995927    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-168.ec2.internal.18c23fb44c042e1c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-168.ec2.internal,UID:ip-10-0-133-168.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-168.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-168.ec2.internal,},FirstTimestamp:2026-07-14 19:47:25.99414326 +0000 UTC m=+0.381918554,LastTimestamp:2026-07-14 19:47:25.99414326 +0000 UTC m=+0.381918554,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-168.ec2.internal,}"
Jul 14 19:47:25.996788 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.996774    2560 policy_none.go:49] "None policy: Start"
Jul 14 19:47:25.996830 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.996793    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 14 19:47:25.996830 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.996804    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 14 19:47:25.998695 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:25.998685    2560 policy_none.go:47] "Start"
Jul 14 19:47:26.032851 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.032836    2560 manager.go:337] "Starting Device Plugin manager"
Jul 14 19:47:26.032916 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:26.032871    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 14 19:47:26.032916 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.032884    2560 server.go:87] "Starting device plugin registration server"
Jul 14 19:47:26.033144 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.033131    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 14 19:47:26.033185 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.033146    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 14 19:47:26.033251 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.033233    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 14 19:47:26.033334 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.033318    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 14 19:47:26.033334 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.033332    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 14 19:47:26.033843 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:26.033825    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 14 19:47:26.033908 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:26.033867    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-168.ec2.internal\" not found"
Jul 14 19:47:26.094332 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.094282    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 14 19:47:26.100633 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.095611    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 14 19:47:26.100633 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.095646    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 14 19:47:26.100633 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.095671    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 14 19:47:26.100633 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:26.095726    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 14 19:47:26.100633 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.098161    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 19:47:26.133240 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.133205    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 19:47:26.134444 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.134426    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 19:47:26.134507 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.134458    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 19:47:26.134507 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.134470    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasSufficientPID"
Jul 14 19:47:26.134507 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.134495    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-168.ec2.internal"
Jul 14 19:47:26.141154 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.141137    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-168.ec2.internal"
Jul 14 19:47:26.141238 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:26.141160    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-168.ec2.internal\": node \"ip-10-0-133-168.ec2.internal\" not found"
Jul 14 19:47:26.154018 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:26.153965    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-168.ec2.internal\" not found"
Jul 14 19:47:26.196333 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.196304    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-168.ec2.internal"]
Jul 14 19:47:26.196466 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.196382    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 19:47:26.197305 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.197288    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 19:47:26.197383 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.197315    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 19:47:26.197383 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.197325    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasSufficientPID"
Jul 14 19:47:26.199591 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.199574    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 19:47:26.199690 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.199676    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal"
Jul 14 19:47:26.199766 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.199724    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 19:47:26.200320 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.200304    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 19:47:26.200320 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.200317    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 19:47:26.200443 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.200333    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 19:47:26.200443 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.200338    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 19:47:26.200443 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.200347    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasSufficientPID"
Jul 14 19:47:26.200443 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.200348    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasSufficientPID"
Jul 14 19:47:26.202607 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.202592    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-168.ec2.internal"
Jul 14 19:47:26.202664 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.202617    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 19:47:26.203392 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.203378    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 19:47:26.203469 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.203401    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 19:47:26.203469 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.203414    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasSufficientPID"
Jul 14 19:47:26.229489 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:26.229463    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-168.ec2.internal\" not found" node="ip-10-0-133-168.ec2.internal"
Jul 14 19:47:26.233803 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:26.233785    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-168.ec2.internal\" not found" node="ip-10-0-133-168.ec2.internal"
Jul 14 19:47:26.254823 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:26.254800    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-168.ec2.internal\" not found"
Jul 14 19:47:26.279271 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.279247    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b9aaaac1f8a2716b7bd3ac221c06559b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal\" (UID: \"b9aaaac1f8a2716b7bd3ac221c06559b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal"
Jul 14 19:47:26.279347 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.279275    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9aaaac1f8a2716b7bd3ac221c06559b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal\" (UID: \"b9aaaac1f8a2716b7bd3ac221c06559b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal"
Jul 14 19:47:26.279347 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.279293    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0536b894286084f9faaec29be192da95-config\") pod \"kube-apiserver-proxy-ip-10-0-133-168.ec2.internal\" (UID: \"0536b894286084f9faaec29be192da95\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-168.ec2.internal"
Jul 14 19:47:26.355669 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:26.355638    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-168.ec2.internal\" not found"
Jul 14 19:47:26.380053 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.380028    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b9aaaac1f8a2716b7bd3ac221c06559b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal\" (UID: \"b9aaaac1f8a2716b7bd3ac221c06559b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal"
Jul 14 19:47:26.380121 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.380060    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9aaaac1f8a2716b7bd3ac221c06559b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal\" (UID: \"b9aaaac1f8a2716b7bd3ac221c06559b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal"
Jul 14 19:47:26.380121 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.380084    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0536b894286084f9faaec29be192da95-config\") pod \"kube-apiserver-proxy-ip-10-0-133-168.ec2.internal\" (UID: \"0536b894286084f9faaec29be192da95\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-168.ec2.internal"
Jul 14 19:47:26.380183 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.380124    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b9aaaac1f8a2716b7bd3ac221c06559b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal\" (UID: \"b9aaaac1f8a2716b7bd3ac221c06559b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal"
Jul 14 19:47:26.380183 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.380133    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0536b894286084f9faaec29be192da95-config\") pod \"kube-apiserver-proxy-ip-10-0-133-168.ec2.internal\" (UID: \"0536b894286084f9faaec29be192da95\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-168.ec2.internal"
Jul 14 19:47:26.380183 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.380129    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9aaaac1f8a2716b7bd3ac221c06559b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal\" (UID: \"b9aaaac1f8a2716b7bd3ac221c06559b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal"
Jul 14 19:47:26.456347 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:26.456279    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-168.ec2.internal\" not found"
Jul 14 19:47:26.530901 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.530866    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal"
Jul 14 19:47:26.536876 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.536859    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-168.ec2.internal"
Jul 14 19:47:26.557051 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:26.557024    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-168.ec2.internal\" not found"
Jul 14 19:47:26.657642 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:26.657599    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-168.ec2.internal\" not found"
Jul 14 19:47:26.758175 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:26.758102    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-168.ec2.internal\" not found"
Jul 14 19:47:26.858689 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:26.858655    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-168.ec2.internal\" not found"
Jul 14 19:47:26.921146 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.921115    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 14 19:47:26.921621 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:26.921244    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 14 19:47:28.746834 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:28.746813    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-168.ec2.internal\" not found"
Jul 14 19:47:28.778574 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:28.778536    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 19:42:27 +0000 UTC" deadline="2028-01-17 18:23:54.942101088 +0000 UTC"
Jul 14 19:47:28.778574 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:28.778562    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13246h36m26.163541867s"
Jul 14 19:47:28.847121 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:28.847046    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-168.ec2.internal\" not found"
Jul 14 19:47:28.904679 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:28.904651    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 19:47:28.947744 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:28.947713    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-168.ec2.internal\" not found"
Jul 14 19:47:29.016662 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.016637    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 19:47:29.048626 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:29.048580    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-168.ec2.internal\" not found"
Jul 14 19:47:29.149186 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:29.149133    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-168.ec2.internal\" not found"
Jul 14 19:47:29.190261 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.190232    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 19:47:29.235295 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.235278    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 19:47:29.264807 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.264785    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-168.ec2.internal"
Jul 14 19:47:29.274582 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.274563    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 14 19:47:29.275439 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.275427    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal"
Jul 14 19:47:29.287916 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.287895    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 14 19:47:29.538784 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.538708    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 19:47:29.747362 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.747328    2560 apiserver.go:52] "Watching apiserver"
Jul 14 19:47:29.754591 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.754569    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 14 19:47:29.754935 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.754910    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-qwdn7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal","openshift-multus/multus-kmfht","openshift-multus/network-metrics-daemon-z2mcj","openshift-network-diagnostics/network-check-target-z785n","kube-system/global-pull-secret-syncer-pp59q","kube-system/kube-apiserver-proxy-ip-10-0-133-168.ec2.internal","openshift-cluster-node-tuning-operator/tuned-9mcfm","openshift-multus/multus-additional-cni-plugins-smrzh","openshift-network-operator/iptables-alerter-rkcxf","openshift-ovn-kubernetes/ovnkube-node-8tzr6","kube-system/konnectivity-agent-xln8v","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw","openshift-dns/node-resolver-bz7rb"]
Jul 14 19:47:29.758574 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.758557    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xln8v"
Jul 14 19:47:29.761819 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.761799    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw"
Jul 14 19:47:29.761925 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.761800    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7svnd\""
Jul 14 19:47:29.762067 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.762035    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 14 19:47:29.762189 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.762156    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 14 19:47:29.764019 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.763999    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rkcxf"
Jul 14 19:47:29.764115 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.764080    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 14 19:47:29.764274 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.764254    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 14 19:47:29.764460 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.764441    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 14 19:47:29.764565 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.764515    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tlgjd\""
Jul 14 19:47:29.764755 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.764742    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 14 19:47:29.766254 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.766234    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:47:29.766341 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:29.766319    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2mcj" podUID="108c0b81-209c-46de-8d27-aba218077646"
Jul 14 19:47:29.766430 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.766415    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 14 19:47:29.766488 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.766453    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 14 19:47:29.766963 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.766949    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 14 19:47:29.766963 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.766955    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tmfjd\""
Jul 14 19:47:29.768527 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.768512    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:47:29.768623 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:29.768599    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z785n" podUID="a11b1aab-78a3-48da-904c-71226e9c85ff"
Jul 14 19:47:29.770595 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.770573    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:29.770679 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:29.770660    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp59q" podUID="120711d8-0534-470b-ab0f-5433b06fef47"
Jul 14 19:47:29.772812 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.772800    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.775039 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.775022    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d979e995-e1cd-4e85-bf27-dd797c072293-device-dir\") pod \"aws-ebs-csi-driver-node-8nhtw\" (UID: \"d979e995-e1cd-4e85-bf27-dd797c072293\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw"
Jul 14 19:47:29.775133 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.775053    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d979e995-e1cd-4e85-bf27-dd797c072293-registration-dir\") pod \"aws-ebs-csi-driver-node-8nhtw\" (UID: \"d979e995-e1cd-4e85-bf27-dd797c072293\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw"
Jul 14 19:47:29.775133 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.775079    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/43a2ff43-5f27-4ad1-8349-28d398a95a63-iptables-alerter-script\") pod \"iptables-alerter-rkcxf\" (UID: \"43a2ff43-5f27-4ad1-8349-28d398a95a63\") " pod="openshift-network-operator/iptables-alerter-rkcxf"
Jul 14 19:47:29.775287 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.775128    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zqpk\" (UniqueName: \"kubernetes.io/projected/108c0b81-209c-46de-8d27-aba218077646-kube-api-access-7zqpk\") pod \"network-metrics-daemon-z2mcj\" (UID: \"108c0b81-209c-46de-8d27-aba218077646\") " pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:47:29.775287 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.775205    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c498088-c01f-4ddd-9c52-0972d109f662-agent-certs\") pod \"konnectivity-agent-xln8v\" (UID: \"6c498088-c01f-4ddd-9c52-0972d109f662\") " pod="kube-system/konnectivity-agent-xln8v"
Jul 14 19:47:29.775287 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.775232    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d979e995-e1cd-4e85-bf27-dd797c072293-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8nhtw\" (UID: \"d979e995-e1cd-4e85-bf27-dd797c072293\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw"
Jul 14 19:47:29.775287 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.775261    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d979e995-e1cd-4e85-bf27-dd797c072293-socket-dir\") pod \"aws-ebs-csi-driver-node-8nhtw\" (UID: \"d979e995-e1cd-4e85-bf27-dd797c072293\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw"
Jul 14 19:47:29.775475 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.775298    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-smrzh"
Jul 14 19:47:29.775475 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.775308    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d979e995-e1cd-4e85-bf27-dd797c072293-sys-fs\") pod \"aws-ebs-csi-driver-node-8nhtw\" (UID: \"d979e995-e1cd-4e85-bf27-dd797c072293\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw"
Jul 14 19:47:29.775475 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.775335    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/108c0b81-209c-46de-8d27-aba218077646-metrics-certs\") pod \"network-metrics-daemon-z2mcj\" (UID: \"108c0b81-209c-46de-8d27-aba218077646\") " pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:47:29.775475 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.775390    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c498088-c01f-4ddd-9c52-0972d109f662-konnectivity-ca\") pod \"konnectivity-agent-xln8v\" (UID: \"6c498088-c01f-4ddd-9c52-0972d109f662\") " pod="kube-system/konnectivity-agent-xln8v"
Jul 14 19:47:29.775475 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.775432    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d979e995-e1cd-4e85-bf27-dd797c072293-etc-selinux\") pod \"aws-ebs-csi-driver-node-8nhtw\" (UID: \"d979e995-e1cd-4e85-bf27-dd797c072293\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw"
Jul 14 19:47:29.775475 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.775456    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rttqh\" (UniqueName: \"kubernetes.io/projected/d979e995-e1cd-4e85-bf27-dd797c072293-kube-api-access-rttqh\") pod \"aws-ebs-csi-driver-node-8nhtw\" (UID: \"d979e995-e1cd-4e85-bf27-dd797c072293\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw"
Jul 14 19:47:29.775755 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.775482    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43a2ff43-5f27-4ad1-8349-28d398a95a63-host-slash\") pod \"iptables-alerter-rkcxf\" (UID: \"43a2ff43-5f27-4ad1-8349-28d398a95a63\") " pod="openshift-network-operator/iptables-alerter-rkcxf"
Jul 14 19:47:29.775755 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.775506    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6d72\" (UniqueName: \"kubernetes.io/projected/43a2ff43-5f27-4ad1-8349-28d398a95a63-kube-api-access-k6d72\") pod \"iptables-alerter-rkcxf\" (UID: \"43a2ff43-5f27-4ad1-8349-28d398a95a63\") " pod="openshift-network-operator/iptables-alerter-rkcxf"
Jul 14 19:47:29.775755 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.775507    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4cnbg\""
Jul 14 19:47:29.775865 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.775778    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 14 19:47:29.775902 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.775880    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 14 19:47:29.777567 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.777549    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 14 19:47:29.777732 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.777714    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qwdn7"
Jul 14 19:47:29.778143 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.778129    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 14 19:47:29.778242 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.778216    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 14 19:47:29.778298 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.778254    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 14 19:47:29.778424 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.778410    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 14 19:47:29.778473 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.778424    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sbmtw\""
Jul 14 19:47:29.778536 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.778520    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 14 19:47:29.780055 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.780035    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.780262 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.780245    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 14 19:47:29.780777 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.780757    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 14 19:47:29.780847 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.780815    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zhmkp\""
Jul 14 19:47:29.780883 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.780815    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 14 19:47:29.782536 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.782519    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.782940 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.782922    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 14 19:47:29.783023 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.782958    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 14 19:47:29.783315 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.783287    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 14 19:47:29.783404 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.783367    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 14 19:47:29.784320 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.784299    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 14 19:47:29.784414 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.784320    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cl9st\""
Jul 14 19:47:29.784768 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.784751    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 14 19:47:29.784979 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.784964    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 14 19:47:29.785278 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.785263    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dm5q6\""
Jul 14 19:47:29.785423 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.785401    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bz7rb"
Jul 14 19:47:29.788075 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.788057    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 14 19:47:29.788075 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.788071    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 14 19:47:29.788230 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.788098    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bg2g8\""
Jul 14 19:47:29.805658 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.805636    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dqfjg"
Jul 14 19:47:29.813559 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.813542    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dqfjg"
Jul 14 19:47:29.866065 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.866039    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 14 19:47:29.876068 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876045    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b5bfcb6a-4113-4273-a3e2-d6b64f2e4937-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-smrzh\" (UID: \"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937\") " pod="openshift-multus/multus-additional-cni-plugins-smrzh"
Jul 14 19:47:29.876219 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876079    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4a36af4c-9eac-465f-9a0e-d65a382530f7-ovnkube-script-lib\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.876219 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876104    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqc9t\" (UniqueName: \"kubernetes.io/projected/4a36af4c-9eac-465f-9a0e-d65a382530f7-kube-api-access-jqc9t\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.876219 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876124    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-system-cni-dir\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.876219 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876144    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-etc-kubernetes\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.876219 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876188    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c498088-c01f-4ddd-9c52-0972d109f662-konnectivity-ca\") pod \"konnectivity-agent-xln8v\" (UID: \"6c498088-c01f-4ddd-9c52-0972d109f662\") " pod="kube-system/konnectivity-agent-xln8v"
Jul 14 19:47:29.876219 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876214    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6d72\" (UniqueName: \"kubernetes.io/projected/43a2ff43-5f27-4ad1-8349-28d398a95a63-kube-api-access-k6d72\") pod \"iptables-alerter-rkcxf\" (UID: \"43a2ff43-5f27-4ad1-8349-28d398a95a63\") " pod="openshift-network-operator/iptables-alerter-rkcxf"
Jul 14 19:47:29.876494 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876238    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-etc-modprobe-d\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.876494 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876261    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-etc-kubernetes\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.876494 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876284    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-etc-systemd\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.876494 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876327    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-host-cni-bin\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.876494 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876360    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-multus-cni-dir\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.876494 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876397    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-etc-sysctl-conf\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.876494 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876418    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-var-lib-kubelet\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.876494 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876433    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5bfcb6a-4113-4273-a3e2-d6b64f2e4937-cnibin\") pod \"multus-additional-cni-plugins-smrzh\" (UID: \"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937\") " pod="openshift-multus/multus-additional-cni-plugins-smrzh"
Jul 14 19:47:29.876494 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876458    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/43a2ff43-5f27-4ad1-8349-28d398a95a63-iptables-alerter-script\") pod \"iptables-alerter-rkcxf\" (UID: \"43a2ff43-5f27-4ad1-8349-28d398a95a63\") " pod="openshift-network-operator/iptables-alerter-rkcxf"
Jul 14 19:47:29.876494 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876475    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-etc-sysconfig\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.876494 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876490    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b5468c6-fd78-4acc-b882-cf389722a101-host\") pod \"node-ca-qwdn7\" (UID: \"1b5468c6-fd78-4acc-b882-cf389722a101\") " pod="openshift-image-registry/node-ca-qwdn7"
Jul 14 19:47:29.877016 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876506    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-host-cni-netd\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.877016 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876521    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-host-var-lib-kubelet\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.877016 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876540    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ced44ef1-063d-48b9-81bc-412dd2a95f74-hosts-file\") pod \"node-resolver-bz7rb\" (UID: \"ced44ef1-063d-48b9-81bc-412dd2a95f74\") " pod="openshift-dns/node-resolver-bz7rb"
Jul 14 19:47:29.877016 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876555    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c498088-c01f-4ddd-9c52-0972d109f662-agent-certs\") pod \"konnectivity-agent-xln8v\" (UID: \"6c498088-c01f-4ddd-9c52-0972d109f662\") " pod="kube-system/konnectivity-agent-xln8v"
Jul 14 19:47:29.877016 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876651    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d979e995-e1cd-4e85-bf27-dd797c072293-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8nhtw\" (UID: \"d979e995-e1cd-4e85-bf27-dd797c072293\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw"
Jul 14 19:47:29.877016 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876686    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.877016 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876716    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a52eed96-0a8b-4a11-99d1-7169c12bfa28-multus-daemon-config\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.877016 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876737    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c498088-c01f-4ddd-9c52-0972d109f662-konnectivity-ca\") pod \"konnectivity-agent-xln8v\" (UID: \"6c498088-c01f-4ddd-9c52-0972d109f662\") " pod="kube-system/konnectivity-agent-xln8v"
Jul 14 19:47:29.877016 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876759    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d979e995-e1cd-4e85-bf27-dd797c072293-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8nhtw\" (UID: \"d979e995-e1cd-4e85-bf27-dd797c072293\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw"
Jul 14 19:47:29.877016 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876790    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d979e995-e1cd-4e85-bf27-dd797c072293-etc-selinux\") pod \"aws-ebs-csi-driver-node-8nhtw\" (UID: \"d979e995-e1cd-4e85-bf27-dd797c072293\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw"
Jul 14 19:47:29.877016 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876812    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-host-slash\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.877016 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876828    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-etc-openvswitch\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.877016 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876851    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-node-log\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.877016 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876886    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqw4j\" (UniqueName: \"kubernetes.io/projected/ced44ef1-063d-48b9-81bc-412dd2a95f74-kube-api-access-dqw4j\") pod \"node-resolver-bz7rb\" (UID: \"ced44ef1-063d-48b9-81bc-412dd2a95f74\") " pod="openshift-dns/node-resolver-bz7rb"
Jul 14 19:47:29.877016 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876889    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d979e995-e1cd-4e85-bf27-dd797c072293-etc-selinux\") pod \"aws-ebs-csi-driver-node-8nhtw\" (UID: \"d979e995-e1cd-4e85-bf27-dd797c072293\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw"
Jul 14 19:47:29.877016 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876908    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d979e995-e1cd-4e85-bf27-dd797c072293-device-dir\") pod \"aws-ebs-csi-driver-node-8nhtw\" (UID: \"d979e995-e1cd-4e85-bf27-dd797c072293\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw"
Jul 14 19:47:29.877016 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876897    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 14 19:47:29.877739 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876926    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-etc-sysctl-d\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.877739 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876950    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhlvv\" (UniqueName: \"kubernetes.io/projected/b5bfcb6a-4113-4273-a3e2-d6b64f2e4937-kube-api-access-nhlvv\") pod \"multus-additional-cni-plugins-smrzh\" (UID: \"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937\") " pod="openshift-multus/multus-additional-cni-plugins-smrzh"
Jul 14 19:47:29.877739 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876965    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-host-var-lib-cni-multus\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.877739 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876982    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zqpk\" (UniqueName: \"kubernetes.io/projected/108c0b81-209c-46de-8d27-aba218077646-kube-api-access-7zqpk\") pod \"network-metrics-daemon-z2mcj\" (UID: \"108c0b81-209c-46de-8d27-aba218077646\") " pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:47:29.877739 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876981    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d979e995-e1cd-4e85-bf27-dd797c072293-device-dir\") pod \"aws-ebs-csi-driver-node-8nhtw\" (UID: \"d979e995-e1cd-4e85-bf27-dd797c072293\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw"
Jul 14 19:47:29.877739 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.876997    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b5bfcb6a-4113-4273-a3e2-d6b64f2e4937-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-smrzh\" (UID: \"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937\") " pod="openshift-multus/multus-additional-cni-plugins-smrzh"
Jul 14 19:47:29.877739 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877012    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gct7\" (UniqueName: \"kubernetes.io/projected/1b5468c6-fd78-4acc-b882-cf389722a101-kube-api-access-4gct7\") pod \"node-ca-qwdn7\" (UID: \"1b5468c6-fd78-4acc-b882-cf389722a101\") " pod="openshift-image-registry/node-ca-qwdn7"
Jul 14 19:47:29.877739 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877030    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-hostroot\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.877739 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877054    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a305099d-5c23-4bad-bd9f-85c0461aa3ef-tmp\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.877739 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877062    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/43a2ff43-5f27-4ad1-8349-28d398a95a63-iptables-alerter-script\") pod \"iptables-alerter-rkcxf\" (UID: \"43a2ff43-5f27-4ad1-8349-28d398a95a63\") " pod="openshift-network-operator/iptables-alerter-rkcxf"
Jul 14 19:47:29.877739 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877069    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-host-run-netns\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.877739 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877117    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-run-openvswitch\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.877739 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877142    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-cnibin\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.877739 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877191    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-host-run-multus-certs\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.877739 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877217    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ced44ef1-063d-48b9-81bc-412dd2a95f74-tmp-dir\") pod \"node-resolver-bz7rb\" (UID: \"ced44ef1-063d-48b9-81bc-412dd2a95f74\") " pod="openshift-dns/node-resolver-bz7rb"
Jul 14 19:47:29.877739 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877241    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-run\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.877739 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877268    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rttqh\" (UniqueName: \"kubernetes.io/projected/d979e995-e1cd-4e85-bf27-dd797c072293-kube-api-access-rttqh\") pod \"aws-ebs-csi-driver-node-8nhtw\" (UID: \"d979e995-e1cd-4e85-bf27-dd797c072293\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw"
Jul 14 19:47:29.878401 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877294    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43a2ff43-5f27-4ad1-8349-28d398a95a63-host-slash\") pod \"iptables-alerter-rkcxf\" (UID: \"43a2ff43-5f27-4ad1-8349-28d398a95a63\") " pod="openshift-network-operator/iptables-alerter-rkcxf"
Jul 14 19:47:29.878401 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877317    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-host-kubelet\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.878401 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877342    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.878401 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877343    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43a2ff43-5f27-4ad1-8349-28d398a95a63-host-slash\") pod \"iptables-alerter-rkcxf\" (UID: \"43a2ff43-5f27-4ad1-8349-28d398a95a63\") " pod="openshift-network-operator/iptables-alerter-rkcxf"
Jul 14 19:47:29.878401 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877365    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a36af4c-9eac-465f-9a0e-d65a382530f7-ovn-node-metrics-cert\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.878401 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877398    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-host\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.878401 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877415    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5bfcb6a-4113-4273-a3e2-d6b64f2e4937-system-cni-dir\") pod \"multus-additional-cni-plugins-smrzh\" (UID: \"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937\") " pod="openshift-multus/multus-additional-cni-plugins-smrzh"
Jul 14 19:47:29.878401 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877431    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-run-ovn\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.878401 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877446    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-multus-conf-dir\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.878401 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877460    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a52eed96-0a8b-4a11-99d1-7169c12bfa28-cni-binary-copy\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.878401 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877484    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d979e995-e1cd-4e85-bf27-dd797c072293-socket-dir\") pod \"aws-ebs-csi-driver-node-8nhtw\" (UID: \"d979e995-e1cd-4e85-bf27-dd797c072293\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw"
Jul 14 19:47:29.878401 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877506    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d979e995-e1cd-4e85-bf27-dd797c072293-sys-fs\") pod \"aws-ebs-csi-driver-node-8nhtw\" (UID: \"d979e995-e1cd-4e85-bf27-dd797c072293\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw"
Jul 14 19:47:29.878401 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877527    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5bfcb6a-4113-4273-a3e2-d6b64f2e4937-tuning-conf-dir\") pod \"multus-additional-cni-plugins-smrzh\" (UID: \"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937\") " pod="openshift-multus/multus-additional-cni-plugins-smrzh"
Jul 14 19:47:29.878401 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877550    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1b5468c6-fd78-4acc-b882-cf389722a101-serviceca\") pod \"node-ca-qwdn7\" (UID: \"1b5468c6-fd78-4acc-b882-cf389722a101\") " pod="openshift-image-registry/node-ca-qwdn7"
Jul 14 19:47:29.878401 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877564    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d979e995-e1cd-4e85-bf27-dd797c072293-sys-fs\") pod \"aws-ebs-csi-driver-node-8nhtw\" (UID: \"d979e995-e1cd-4e85-bf27-dd797c072293\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw"
Jul 14 19:47:29.878401 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877572    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-run-systemd\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.878401 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877573    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d979e995-e1cd-4e85-bf27-dd797c072293-socket-dir\") pod \"aws-ebs-csi-driver-node-8nhtw\" (UID: \"d979e995-e1cd-4e85-bf27-dd797c072293\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw"
Jul 14 19:47:29.878876 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877596    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2f9f\" (UniqueName: \"kubernetes.io/projected/a52eed96-0a8b-4a11-99d1-7169c12bfa28-kube-api-access-n2f9f\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.878876 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877616    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncmfq\" (UniqueName: \"kubernetes.io/projected/a11b1aab-78a3-48da-904c-71226e9c85ff-kube-api-access-ncmfq\") pod \"network-check-target-z785n\" (UID: \"a11b1aab-78a3-48da-904c-71226e9c85ff\") " pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:47:29.878876 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877631    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a305099d-5c23-4bad-bd9f-85c0461aa3ef-etc-tuned\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.878876 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877645    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b5bfcb6a-4113-4273-a3e2-d6b64f2e4937-os-release\") pod \"multus-additional-cni-plugins-smrzh\" (UID: \"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937\") " pod="openshift-multus/multus-additional-cni-plugins-smrzh"
Jul 14 19:47:29.878876 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877663    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-log-socket\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.878876 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877684    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4a36af4c-9eac-465f-9a0e-d65a382530f7-ovnkube-config\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.878876 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877705    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-multus-socket-dir-parent\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.878876 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877739    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-host-run-k8s-cni-cncf-io\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.878876 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877768    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/120711d8-0534-470b-ab0f-5433b06fef47-dbus\") pod \"global-pull-secret-syncer-pp59q\" (UID: \"120711d8-0534-470b-ab0f-5433b06fef47\") " pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:29.878876 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877782    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-sys\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.878876 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877798    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5hjz\" (UniqueName: \"kubernetes.io/projected/a305099d-5c23-4bad-bd9f-85c0461aa3ef-kube-api-access-x5hjz\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.878876 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877813    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-systemd-units\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.878876 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877828    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4a36af4c-9eac-465f-9a0e-d65a382530f7-env-overrides\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.878876 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877844    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d979e995-e1cd-4e85-bf27-dd797c072293-registration-dir\") pod \"aws-ebs-csi-driver-node-8nhtw\" (UID: \"d979e995-e1cd-4e85-bf27-dd797c072293\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw"
Jul 14 19:47:29.878876 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877859    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/120711d8-0534-470b-ab0f-5433b06fef47-original-pull-secret\") pod \"global-pull-secret-syncer-pp59q\" (UID: \"120711d8-0534-470b-ab0f-5433b06fef47\") " pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:29.878876 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877874    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-lib-modules\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.879336 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877891    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5bfcb6a-4113-4273-a3e2-d6b64f2e4937-cni-binary-copy\") pod \"multus-additional-cni-plugins-smrzh\" (UID: \"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937\") " pod="openshift-multus/multus-additional-cni-plugins-smrzh"
Jul 14 19:47:29.879336 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877905    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-var-lib-openvswitch\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.879336 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877923    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-os-release\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.879336 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877940    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-host-run-netns\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.879336 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877963    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-host-var-lib-cni-bin\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.879336 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877986    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/108c0b81-209c-46de-8d27-aba218077646-metrics-certs\") pod \"network-metrics-daemon-z2mcj\" (UID: \"108c0b81-209c-46de-8d27-aba218077646\") " pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:47:29.879336 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.877987    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d979e995-e1cd-4e85-bf27-dd797c072293-registration-dir\") pod \"aws-ebs-csi-driver-node-8nhtw\" (UID: \"d979e995-e1cd-4e85-bf27-dd797c072293\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw"
Jul 14 19:47:29.879336 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.878008    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/120711d8-0534-470b-ab0f-5433b06fef47-kubelet-config\") pod \"global-pull-secret-syncer-pp59q\" (UID: \"120711d8-0534-470b-ab0f-5433b06fef47\") " pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:29.879336 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:29.878087    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 19:47:29.879336 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:29.878161    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/108c0b81-209c-46de-8d27-aba218077646-metrics-certs podName:108c0b81-209c-46de-8d27-aba218077646 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:30.378143093 +0000 UTC m=+2.978446790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/108c0b81-209c-46de-8d27-aba218077646-metrics-certs") pod "network-metrics-daemon-z2mcj" (UID: "108c0b81-209c-46de-8d27-aba218077646") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 19:47:29.880135 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.880119    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c498088-c01f-4ddd-9c52-0972d109f662-agent-certs\") pod \"konnectivity-agent-xln8v\" (UID: \"6c498088-c01f-4ddd-9c52-0972d109f662\") " pod="kube-system/konnectivity-agent-xln8v"
Jul 14 19:47:29.887649 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.887625    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6d72\" (UniqueName: \"kubernetes.io/projected/43a2ff43-5f27-4ad1-8349-28d398a95a63-kube-api-access-k6d72\") pod \"iptables-alerter-rkcxf\" (UID: \"43a2ff43-5f27-4ad1-8349-28d398a95a63\") " pod="openshift-network-operator/iptables-alerter-rkcxf"
Jul 14 19:47:29.887812 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.887780    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-168.ec2.internal" event={"ID":"0536b894286084f9faaec29be192da95","Type":"ContainerStarted","Data":"b87656d62838332e9386c866bf87e28b710857042fc73cd52e9bd1d27e87e1bd"}
Jul 14 19:47:29.888607 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.888588    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rttqh\" (UniqueName: \"kubernetes.io/projected/d979e995-e1cd-4e85-bf27-dd797c072293-kube-api-access-rttqh\") pod \"aws-ebs-csi-driver-node-8nhtw\" (UID: \"d979e995-e1cd-4e85-bf27-dd797c072293\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw"
Jul 14 19:47:29.889036 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.889019    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zqpk\" (UniqueName: \"kubernetes.io/projected/108c0b81-209c-46de-8d27-aba218077646-kube-api-access-7zqpk\") pod \"network-metrics-daemon-z2mcj\" (UID: \"108c0b81-209c-46de-8d27-aba218077646\") " pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:47:29.978222 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.978157    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-host-var-lib-kubelet\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.978222 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.978221    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ced44ef1-063d-48b9-81bc-412dd2a95f74-hosts-file\") pod \"node-resolver-bz7rb\" (UID: \"ced44ef1-063d-48b9-81bc-412dd2a95f74\") " pod="openshift-dns/node-resolver-bz7rb"
Jul 14 19:47:29.978450 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.978242    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.978450 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.978263    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a52eed96-0a8b-4a11-99d1-7169c12bfa28-multus-daemon-config\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.978450 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.978285    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-host-var-lib-kubelet\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.978450 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.978304    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.978450 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.978340    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-host-slash\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.978450 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.978368    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ced44ef1-063d-48b9-81bc-412dd2a95f74-hosts-file\") pod \"node-resolver-bz7rb\" (UID: \"ced44ef1-063d-48b9-81bc-412dd2a95f74\") " pod="openshift-dns/node-resolver-bz7rb"
Jul 14 19:47:29.978450 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.978379    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-host-slash\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.978450 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.978376    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-etc-openvswitch\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.978450 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.978407    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-etc-openvswitch\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.978450 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.978420    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-node-log\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.978450 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.978441    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-node-log\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.978450 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.978446    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqw4j\" (UniqueName: \"kubernetes.io/projected/ced44ef1-063d-48b9-81bc-412dd2a95f74-kube-api-access-dqw4j\") pod \"node-resolver-bz7rb\" (UID: \"ced44ef1-063d-48b9-81bc-412dd2a95f74\") " pod="openshift-dns/node-resolver-bz7rb"
Jul 14 19:47:29.978960 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.978474    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-etc-sysctl-d\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.978960 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.978501    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhlvv\" (UniqueName: \"kubernetes.io/projected/b5bfcb6a-4113-4273-a3e2-d6b64f2e4937-kube-api-access-nhlvv\") pod \"multus-additional-cni-plugins-smrzh\" (UID: \"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937\") " pod="openshift-multus/multus-additional-cni-plugins-smrzh"
Jul 14 19:47:29.978960 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.978526    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-host-var-lib-cni-multus\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.978960 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.978555    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b5bfcb6a-4113-4273-a3e2-d6b64f2e4937-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-smrzh\" (UID: \"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937\") " pod="openshift-multus/multus-additional-cni-plugins-smrzh"
Jul 14 19:47:29.978960 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.978626    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-host-var-lib-cni-multus\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.979211 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.979097    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a52eed96-0a8b-4a11-99d1-7169c12bfa28-multus-daemon-config\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.979324 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.979209    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-etc-sysctl-d\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.979324 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.978661    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gct7\" (UniqueName: \"kubernetes.io/projected/1b5468c6-fd78-4acc-b882-cf389722a101-kube-api-access-4gct7\") pod \"node-ca-qwdn7\" (UID: \"1b5468c6-fd78-4acc-b882-cf389722a101\") " pod="openshift-image-registry/node-ca-qwdn7"
Jul 14 19:47:29.979324 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.979287    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-hostroot\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.979324 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.979314    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a305099d-5c23-4bad-bd9f-85c0461aa3ef-tmp\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.979497 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.979348    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-host-run-netns\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.979497 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.979366    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-hostroot\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.979497 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.979378    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-run-openvswitch\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.979497 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.979423    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-run-openvswitch\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.979497 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.979424    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-cnibin\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.979497 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.979475    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-cnibin\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.979748 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.979517    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-host-run-netns\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.979748 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.979712    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b5bfcb6a-4113-4273-a3e2-d6b64f2e4937-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-smrzh\" (UID: \"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937\") " pod="openshift-multus/multus-additional-cni-plugins-smrzh"
Jul 14 19:47:29.980008 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.979985    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-host-run-multus-certs\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.980075 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980037    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ced44ef1-063d-48b9-81bc-412dd2a95f74-tmp-dir\") pod \"node-resolver-bz7rb\" (UID: \"ced44ef1-063d-48b9-81bc-412dd2a95f74\") " pod="openshift-dns/node-resolver-bz7rb"
Jul 14 19:47:29.980075 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980068    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-run\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.980191 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980093    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-host-kubelet\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.980191 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980123    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.980191 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980151    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a36af4c-9eac-465f-9a0e-d65a382530f7-ovn-node-metrics-cert\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.980329 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980199    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-host\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.980329 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980226    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5bfcb6a-4113-4273-a3e2-d6b64f2e4937-system-cni-dir\") pod \"multus-additional-cni-plugins-smrzh\" (UID: \"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937\") " pod="openshift-multus/multus-additional-cni-plugins-smrzh"
Jul 14 19:47:29.980329 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980257    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-run-ovn\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.980329 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980287    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-multus-conf-dir\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.980329 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980316    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a52eed96-0a8b-4a11-99d1-7169c12bfa28-cni-binary-copy\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.980537 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980338    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5bfcb6a-4113-4273-a3e2-d6b64f2e4937-tuning-conf-dir\") pod \"multus-additional-cni-plugins-smrzh\" (UID: \"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937\") " pod="openshift-multus/multus-additional-cni-plugins-smrzh"
Jul 14 19:47:29.980537 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980357    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1b5468c6-fd78-4acc-b882-cf389722a101-serviceca\") pod \"node-ca-qwdn7\" (UID: \"1b5468c6-fd78-4acc-b882-cf389722a101\") " pod="openshift-image-registry/node-ca-qwdn7"
Jul 14 19:47:29.980537 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980373    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-run-systemd\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.980537 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980404    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2f9f\" (UniqueName: \"kubernetes.io/projected/a52eed96-0a8b-4a11-99d1-7169c12bfa28-kube-api-access-n2f9f\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.980537 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980434    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncmfq\" (UniqueName: \"kubernetes.io/projected/a11b1aab-78a3-48da-904c-71226e9c85ff-kube-api-access-ncmfq\") pod \"network-check-target-z785n\" (UID: \"a11b1aab-78a3-48da-904c-71226e9c85ff\") " pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:47:29.980537 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980454    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a305099d-5c23-4bad-bd9f-85c0461aa3ef-etc-tuned\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.980537 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980470    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b5bfcb6a-4113-4273-a3e2-d6b64f2e4937-os-release\") pod \"multus-additional-cni-plugins-smrzh\" (UID: \"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937\") " pod="openshift-multus/multus-additional-cni-plugins-smrzh"
Jul 14 19:47:29.980537 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980477    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-host-run-multus-certs\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.980537 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980496    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-log-socket\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.980913 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980538    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4a36af4c-9eac-465f-9a0e-d65a382530f7-ovnkube-config\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.980913 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980547    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5bfcb6a-4113-4273-a3e2-d6b64f2e4937-system-cni-dir\") pod \"multus-additional-cni-plugins-smrzh\" (UID: \"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937\") " pod="openshift-multus/multus-additional-cni-plugins-smrzh"
Jul 14 19:47:29.980913 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980582    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-multus-socket-dir-parent\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.980913 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980597    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-run\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.980913 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980604    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-host-run-k8s-cni-cncf-io\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.980913 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980623    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/120711d8-0534-470b-ab0f-5433b06fef47-dbus\") pod \"global-pull-secret-syncer-pp59q\" (UID: \"120711d8-0534-470b-ab0f-5433b06fef47\") " pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:29.980913 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980648    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-host-kubelet\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.980913 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980664    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-sys\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.980913 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980686    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5hjz\" (UniqueName: \"kubernetes.io/projected/a305099d-5c23-4bad-bd9f-85c0461aa3ef-kube-api-access-x5hjz\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.980913 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980693    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.980913 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980704    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-systemd-units\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.980913 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980739    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4a36af4c-9eac-465f-9a0e-d65a382530f7-env-overrides\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.980913 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980760    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/120711d8-0534-470b-ab0f-5433b06fef47-original-pull-secret\") pod \"global-pull-secret-syncer-pp59q\" (UID: \"120711d8-0534-470b-ab0f-5433b06fef47\") " pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:29.980913 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980780    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-lib-modules\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.980913 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980806    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5bfcb6a-4113-4273-a3e2-d6b64f2e4937-cni-binary-copy\") pod \"multus-additional-cni-plugins-smrzh\" (UID: \"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937\") " pod="openshift-multus/multus-additional-cni-plugins-smrzh"
Jul 14 19:47:29.980913 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980824    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-var-lib-openvswitch\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.980913 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980846    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-os-release\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.981656 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980864    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-host-run-netns\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.981656 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980890    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-host-var-lib-cni-bin\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.981656 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980921    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/120711d8-0534-470b-ab0f-5433b06fef47-kubelet-config\") pod \"global-pull-secret-syncer-pp59q\" (UID: \"120711d8-0534-470b-ab0f-5433b06fef47\") " pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:29.981656 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980940    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b5bfcb6a-4113-4273-a3e2-d6b64f2e4937-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-smrzh\" (UID: \"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937\") " pod="openshift-multus/multus-additional-cni-plugins-smrzh"
Jul 14 19:47:29.981656 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.980968    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4a36af4c-9eac-465f-9a0e-d65a382530f7-ovnkube-script-lib\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.981656 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.981000    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqc9t\" (UniqueName: \"kubernetes.io/projected/4a36af4c-9eac-465f-9a0e-d65a382530f7-kube-api-access-jqc9t\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.981656 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.981032    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-system-cni-dir\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.981656 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.981058    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-etc-kubernetes\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.981656 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.981091    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-etc-modprobe-d\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.981656 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.981123    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-etc-kubernetes\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.981656 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.981155    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-etc-systemd\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.981656 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.981205    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-host-cni-bin\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.981656 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.981231    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-multus-cni-dir\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.981656 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.981262    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-etc-sysctl-conf\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.981656 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.981293    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-var-lib-kubelet\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.981656 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.981326    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5bfcb6a-4113-4273-a3e2-d6b64f2e4937-cnibin\") pod \"multus-additional-cni-plugins-smrzh\" (UID: \"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937\") " pod="openshift-multus/multus-additional-cni-plugins-smrzh"
Jul 14 19:47:29.981656 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.981352    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-etc-sysconfig\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.982397 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.981427    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-etc-sysconfig\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.982397 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.981529    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-etc-kubernetes\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.982397 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.981593    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-lib-modules\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.982397 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.981637    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-etc-modprobe-d\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.982397 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.981670    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-system-cni-dir\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.982397 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.981695    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-etc-kubernetes\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.982397 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:29.981732    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 19:47:29.982397 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.981755    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-etc-systemd\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.982397 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:29.981797    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/120711d8-0534-470b-ab0f-5433b06fef47-original-pull-secret podName:120711d8-0534-470b-ab0f-5433b06fef47 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:30.481773398 +0000 UTC m=+3.082077093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/120711d8-0534-470b-ab0f-5433b06fef47-original-pull-secret") pod "global-pull-secret-syncer-pp59q" (UID: "120711d8-0534-470b-ab0f-5433b06fef47") : object "kube-system"/"original-pull-secret" not registered
Jul 14 19:47:29.982397 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.981804    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-host-cni-bin\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.982397 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.981802    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4a36af4c-9eac-465f-9a0e-d65a382530f7-env-overrides\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.982397 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.981827    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b5468c6-fd78-4acc-b882-cf389722a101-host\") pod \"node-ca-qwdn7\" (UID: \"1b5468c6-fd78-4acc-b882-cf389722a101\") " pod="openshift-image-registry/node-ca-qwdn7"
Jul 14 19:47:29.982397 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.981866    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-host-cni-netd\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.982397 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.981922    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-host-var-lib-cni-bin\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.982397 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.981975    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-run-ovn\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.982397 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.982387    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1b5468c6-fd78-4acc-b882-cf389722a101-serviceca\") pod \"node-ca-qwdn7\" (UID: \"1b5468c6-fd78-4acc-b882-cf389722a101\") " pod="openshift-image-registry/node-ca-qwdn7"
Jul 14 19:47:29.983100 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.982458    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-run-systemd\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.983100 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.982532    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/120711d8-0534-470b-ab0f-5433b06fef47-kubelet-config\") pod \"global-pull-secret-syncer-pp59q\" (UID: \"120711d8-0534-470b-ab0f-5433b06fef47\") " pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:29.983100 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.982575    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4a36af4c-9eac-465f-9a0e-d65a382530f7-ovnkube-script-lib\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.983100 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.982812    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-multus-cni-dir\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.983100 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.982881    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-os-release\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.983100 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.982976    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-etc-sysctl-conf\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.983100 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.983043    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b5bfcb6a-4113-4273-a3e2-d6b64f2e4937-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-smrzh\" (UID: \"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937\") " pod="openshift-multus/multus-additional-cni-plugins-smrzh"
Jul 14 19:47:29.983441 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.983107    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5bfcb6a-4113-4273-a3e2-d6b64f2e4937-cnibin\") pod \"multus-additional-cni-plugins-smrzh\" (UID: \"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937\") " pod="openshift-multus/multus-additional-cni-plugins-smrzh"
Jul 14 19:47:29.983441 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.983129    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5bfcb6a-4113-4273-a3e2-d6b64f2e4937-tuning-conf-dir\") pod \"multus-additional-cni-plugins-smrzh\" (UID: \"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937\") " pod="openshift-multus/multus-additional-cni-plugins-smrzh"
Jul 14 19:47:29.983441 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.983161    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-host-run-k8s-cni-cncf-io\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.983441 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.983206    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-host\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.983441 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.983325    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-host-run-netns\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.983441 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.983349    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-host-cni-netd\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.983441 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.981975    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-multus-conf-dir\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.983441 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.983381    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/120711d8-0534-470b-ab0f-5433b06fef47-dbus\") pod \"global-pull-secret-syncer-pp59q\" (UID: \"120711d8-0534-470b-ab0f-5433b06fef47\") " pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:29.983874 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.983474    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-sys\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.983874 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.983429    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-systemd-units\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.983874 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.983659    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a52eed96-0a8b-4a11-99d1-7169c12bfa28-multus-socket-dir-parent\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.983874 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.983655    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b5bfcb6a-4113-4273-a3e2-d6b64f2e4937-os-release\") pod \"multus-additional-cni-plugins-smrzh\" (UID: \"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937\") " pod="openshift-multus/multus-additional-cni-plugins-smrzh"
Jul 14 19:47:29.983874 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.983699    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b5468c6-fd78-4acc-b882-cf389722a101-host\") pod \"node-ca-qwdn7\" (UID: \"1b5468c6-fd78-4acc-b882-cf389722a101\") " pod="openshift-image-registry/node-ca-qwdn7"
Jul 14 19:47:29.983874 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.983803    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ced44ef1-063d-48b9-81bc-412dd2a95f74-tmp-dir\") pod \"node-resolver-bz7rb\" (UID: \"ced44ef1-063d-48b9-81bc-412dd2a95f74\") " pod="openshift-dns/node-resolver-bz7rb"
Jul 14 19:47:29.984206 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.983893    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-log-socket\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.984206 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.983937    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a36af4c-9eac-465f-9a0e-d65a382530f7-var-lib-openvswitch\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.984206 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.983939    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5bfcb6a-4113-4273-a3e2-d6b64f2e4937-cni-binary-copy\") pod \"multus-additional-cni-plugins-smrzh\" (UID: \"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937\") " pod="openshift-multus/multus-additional-cni-plugins-smrzh"
Jul 14 19:47:29.984206 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.984048    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a305099d-5c23-4bad-bd9f-85c0461aa3ef-var-lib-kubelet\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.984206 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.984160    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4a36af4c-9eac-465f-9a0e-d65a382530f7-ovnkube-config\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.984438 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.984277    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a52eed96-0a8b-4a11-99d1-7169c12bfa28-cni-binary-copy\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.984438 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.984363    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a305099d-5c23-4bad-bd9f-85c0461aa3ef-tmp\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.985299 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.985278    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a305099d-5c23-4bad-bd9f-85c0461aa3ef-etc-tuned\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:29.985609 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.985591    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a36af4c-9eac-465f-9a0e-d65a382530f7-ovn-node-metrics-cert\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.988109 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.988082    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gct7\" (UniqueName: \"kubernetes.io/projected/1b5468c6-fd78-4acc-b882-cf389722a101-kube-api-access-4gct7\") pod \"node-ca-qwdn7\" (UID: \"1b5468c6-fd78-4acc-b882-cf389722a101\") " pod="openshift-image-registry/node-ca-qwdn7"
Jul 14 19:47:29.988205 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.988138    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhlvv\" (UniqueName: \"kubernetes.io/projected/b5bfcb6a-4113-4273-a3e2-d6b64f2e4937-kube-api-access-nhlvv\") pod \"multus-additional-cni-plugins-smrzh\" (UID: \"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937\") " pod="openshift-multus/multus-additional-cni-plugins-smrzh"
Jul 14 19:47:29.988253 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.988238    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqw4j\" (UniqueName: \"kubernetes.io/projected/ced44ef1-063d-48b9-81bc-412dd2a95f74-kube-api-access-dqw4j\") pod \"node-resolver-bz7rb\" (UID: \"ced44ef1-063d-48b9-81bc-412dd2a95f74\") " pod="openshift-dns/node-resolver-bz7rb"
Jul 14 19:47:29.989476 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:29.989455    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 19:47:29.989476 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:29.989481    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 19:47:29.989609 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:29.989494    2560 projected.go:196] Error preparing data for projected volume kube-api-access-ncmfq for pod openshift-network-diagnostics/network-check-target-z785n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 19:47:29.989609 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:29.989560    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a11b1aab-78a3-48da-904c-71226e9c85ff-kube-api-access-ncmfq podName:a11b1aab-78a3-48da-904c-71226e9c85ff nodeName:}" failed. No retries permitted until 2026-07-14 19:47:30.489543753 +0000 UTC m=+3.089847448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ncmfq" (UniqueName: "kubernetes.io/projected/a11b1aab-78a3-48da-904c-71226e9c85ff-kube-api-access-ncmfq") pod "network-check-target-z785n" (UID: "a11b1aab-78a3-48da-904c-71226e9c85ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 19:47:29.990685 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.990663    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqc9t\" (UniqueName: \"kubernetes.io/projected/4a36af4c-9eac-465f-9a0e-d65a382530f7-kube-api-access-jqc9t\") pod \"ovnkube-node-8tzr6\" (UID: \"4a36af4c-9eac-465f-9a0e-d65a382530f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:29.991467 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.991451    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2f9f\" (UniqueName: \"kubernetes.io/projected/a52eed96-0a8b-4a11-99d1-7169c12bfa28-kube-api-access-n2f9f\") pod \"multus-kmfht\" (UID: \"a52eed96-0a8b-4a11-99d1-7169c12bfa28\") " pod="openshift-multus/multus-kmfht"
Jul 14 19:47:29.991747 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:29.991728    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5hjz\" (UniqueName: \"kubernetes.io/projected/a305099d-5c23-4bad-bd9f-85c0461aa3ef-kube-api-access-x5hjz\") pod \"tuned-9mcfm\" (UID: \"a305099d-5c23-4bad-bd9f-85c0461aa3ef\") " pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:30.071286 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:30.071195    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xln8v"
Jul 14 19:47:30.076520 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:30.076500    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw"
Jul 14 19:47:30.081411 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:30.081384    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c498088_c01f_4ddd_9c52_0972d109f662.slice/crio-bc7e673a09c0ccdb4f4866faa2a046a566a75d3983a2d5b69cbc75387e6f4073 WatchSource:0}: Error finding container bc7e673a09c0ccdb4f4866faa2a046a566a75d3983a2d5b69cbc75387e6f4073: Status 404 returned error can't find the container with id bc7e673a09c0ccdb4f4866faa2a046a566a75d3983a2d5b69cbc75387e6f4073
Jul 14 19:47:30.083550 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:30.083532    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rkcxf"
Jul 14 19:47:30.085835 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:30.085809    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd979e995_e1cd_4e85_bf27_dd797c072293.slice/crio-22f9c686aad51e18a522bf67803b1b02077e566c6b7ebf0f959840daa5c79562 WatchSource:0}: Error finding container 22f9c686aad51e18a522bf67803b1b02077e566c6b7ebf0f959840daa5c79562: Status 404 returned error can't find the container with id 22f9c686aad51e18a522bf67803b1b02077e566c6b7ebf0f959840daa5c79562
Jul 14 19:47:30.088463 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:30.088447    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9mcfm"
Jul 14 19:47:30.094428 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:30.094262    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-smrzh"
Jul 14 19:47:30.094597 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:30.094561    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a2ff43_5f27_4ad1_8349_28d398a95a63.slice/crio-9fc7ec000505c056fb6666c7d92e4376b473b563f8e4d6991d477fd6500767d5 WatchSource:0}: Error finding container 9fc7ec000505c056fb6666c7d92e4376b473b563f8e4d6991d477fd6500767d5: Status 404 returned error can't find the container with id 9fc7ec000505c056fb6666c7d92e4376b473b563f8e4d6991d477fd6500767d5
Jul 14 19:47:30.100909 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:30.100886    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qwdn7"
Jul 14 19:47:30.101514 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:30.101491    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda305099d_5c23_4bad_bd9f_85c0461aa3ef.slice/crio-f56bd5dccc5a6bcfdbcb9785cf75d29be35b34051fcd77f82c346b86610987c0 WatchSource:0}: Error finding container f56bd5dccc5a6bcfdbcb9785cf75d29be35b34051fcd77f82c346b86610987c0: Status 404 returned error can't find the container with id f56bd5dccc5a6bcfdbcb9785cf75d29be35b34051fcd77f82c346b86610987c0
Jul 14 19:47:30.107243 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:30.104488    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5bfcb6a_4113_4273_a3e2_d6b64f2e4937.slice/crio-45490f85902a150c2c97ac804a6b15d498072b5861f0ecd3f22f83331f4daeb9 WatchSource:0}: Error finding container 45490f85902a150c2c97ac804a6b15d498072b5861f0ecd3f22f83331f4daeb9: Status 404 returned error can't find the container with id 45490f85902a150c2c97ac804a6b15d498072b5861f0ecd3f22f83331f4daeb9
Jul 14 19:47:30.107243 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:30.105533    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:30.110652 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:30.110631    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b5468c6_fd78_4acc_b882_cf389722a101.slice/crio-b8aca118c8492d56e4c3cbafab5f9f9eb70c16e901641d2cd9e639db9927c322 WatchSource:0}: Error finding container b8aca118c8492d56e4c3cbafab5f9f9eb70c16e901641d2cd9e639db9927c322: Status 404 returned error can't find the container with id b8aca118c8492d56e4c3cbafab5f9f9eb70c16e901641d2cd9e639db9927c322
Jul 14 19:47:30.110732 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:30.110669    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kmfht"
Jul 14 19:47:30.115623 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:30.115604    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bz7rb"
Jul 14 19:47:30.115886 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:30.115869    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a36af4c_9eac_465f_9a0e_d65a382530f7.slice/crio-4e35dc16909c141ecf0b1f63ab2873003411960f8b2bbeb608d2aae8da3b80a6 WatchSource:0}: Error finding container 4e35dc16909c141ecf0b1f63ab2873003411960f8b2bbeb608d2aae8da3b80a6: Status 404 returned error can't find the container with id 4e35dc16909c141ecf0b1f63ab2873003411960f8b2bbeb608d2aae8da3b80a6
Jul 14 19:47:30.120892 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:30.120862    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda52eed96_0a8b_4a11_99d1_7169c12bfa28.slice/crio-5e4e3b8312c17d74276c788167fd973f9d32eefb3e188e82ea69a0dd3aea7488 WatchSource:0}: Error finding container 5e4e3b8312c17d74276c788167fd973f9d32eefb3e188e82ea69a0dd3aea7488: Status 404 returned error can't find the container with id 5e4e3b8312c17d74276c788167fd973f9d32eefb3e188e82ea69a0dd3aea7488
Jul 14 19:47:30.125485 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:30.125466    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced44ef1_063d_48b9_81bc_412dd2a95f74.slice/crio-0fd0738453ff03313fdf678fb04a7ca91e057007c1db22fd2681e879ba62104b WatchSource:0}: Error finding container 0fd0738453ff03313fdf678fb04a7ca91e057007c1db22fd2681e879ba62104b: Status 404 returned error can't find the container with id 0fd0738453ff03313fdf678fb04a7ca91e057007c1db22fd2681e879ba62104b
Jul 14 19:47:30.170095 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:47:30.170072    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9aaaac1f8a2716b7bd3ac221c06559b.slice/crio-fef4620dd1e367edec7a535de07ba0d9184ef87692b5d41f795a72ff3728cea6 WatchSource:0}: Error finding container fef4620dd1e367edec7a535de07ba0d9184ef87692b5d41f795a72ff3728cea6: Status 404 returned error can't find the container with id fef4620dd1e367edec7a535de07ba0d9184ef87692b5d41f795a72ff3728cea6
Jul 14 19:47:30.384790 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:30.384711    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/108c0b81-209c-46de-8d27-aba218077646-metrics-certs\") pod \"network-metrics-daemon-z2mcj\" (UID: \"108c0b81-209c-46de-8d27-aba218077646\") " pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:47:30.384953 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:30.384830    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 19:47:30.384953 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:30.384888    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/108c0b81-209c-46de-8d27-aba218077646-metrics-certs podName:108c0b81-209c-46de-8d27-aba218077646 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:31.384871073 +0000 UTC m=+3.985174754 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/108c0b81-209c-46de-8d27-aba218077646-metrics-certs") pod "network-metrics-daemon-z2mcj" (UID: "108c0b81-209c-46de-8d27-aba218077646") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 19:47:30.486378 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:30.485792    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/120711d8-0534-470b-ab0f-5433b06fef47-original-pull-secret\") pod \"global-pull-secret-syncer-pp59q\" (UID: \"120711d8-0534-470b-ab0f-5433b06fef47\") " pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:30.486378 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:30.485954    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 19:47:30.486378 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:30.486016    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/120711d8-0534-470b-ab0f-5433b06fef47-original-pull-secret podName:120711d8-0534-470b-ab0f-5433b06fef47 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:31.485996183 +0000 UTC m=+4.086299867 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/120711d8-0534-470b-ab0f-5433b06fef47-original-pull-secret") pod "global-pull-secret-syncer-pp59q" (UID: "120711d8-0534-470b-ab0f-5433b06fef47") : object "kube-system"/"original-pull-secret" not registered
Jul 14 19:47:30.586475 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:30.586440    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncmfq\" (UniqueName: \"kubernetes.io/projected/a11b1aab-78a3-48da-904c-71226e9c85ff-kube-api-access-ncmfq\") pod \"network-check-target-z785n\" (UID: \"a11b1aab-78a3-48da-904c-71226e9c85ff\") " pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:47:30.586667 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:30.586649    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 19:47:30.586705 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:30.586675    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 19:47:30.586705 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:30.586688    2560 projected.go:196] Error preparing data for projected volume kube-api-access-ncmfq for pod openshift-network-diagnostics/network-check-target-z785n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 19:47:30.586765 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:30.586754    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a11b1aab-78a3-48da-904c-71226e9c85ff-kube-api-access-ncmfq podName:a11b1aab-78a3-48da-904c-71226e9c85ff nodeName:}" failed. No retries permitted until 2026-07-14 19:47:31.586724548 +0000 UTC m=+4.187028234 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ncmfq" (UniqueName: "kubernetes.io/projected/a11b1aab-78a3-48da-904c-71226e9c85ff-kube-api-access-ncmfq") pod "network-check-target-z785n" (UID: "a11b1aab-78a3-48da-904c-71226e9c85ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 19:47:30.817385 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:30.817240    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 19:42:29 +0000 UTC" deadline="2027-12-20 11:53:26.320643075 +0000 UTC"
Jul 14 19:47:30.817385 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:30.817278    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12568h5m55.503369459s"
Jul 14 19:47:30.883676 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:30.883644    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:47:30.883834 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:30.883770    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z785n" podUID="a11b1aab-78a3-48da-904c-71226e9c85ff"
Jul 14 19:47:30.884270 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:30.884246    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:47:30.884397 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:30.884376    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2mcj" podUID="108c0b81-209c-46de-8d27-aba218077646"
Jul 14 19:47:30.897965 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:30.897931    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bz7rb" event={"ID":"ced44ef1-063d-48b9-81bc-412dd2a95f74","Type":"ContainerStarted","Data":"0fd0738453ff03313fdf678fb04a7ca91e057007c1db22fd2681e879ba62104b"}
Jul 14 19:47:30.899529 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:30.899502    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9mcfm" event={"ID":"a305099d-5c23-4bad-bd9f-85c0461aa3ef","Type":"ContainerStarted","Data":"f56bd5dccc5a6bcfdbcb9785cf75d29be35b34051fcd77f82c346b86610987c0"}
Jul 14 19:47:30.902065 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:30.902039    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rkcxf" event={"ID":"43a2ff43-5f27-4ad1-8349-28d398a95a63","Type":"ContainerStarted","Data":"9fc7ec000505c056fb6666c7d92e4376b473b563f8e4d6991d477fd6500767d5"}
Jul 14 19:47:30.906576 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:30.906546    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw" event={"ID":"d979e995-e1cd-4e85-bf27-dd797c072293","Type":"ContainerStarted","Data":"22f9c686aad51e18a522bf67803b1b02077e566c6b7ebf0f959840daa5c79562"}
Jul 14 19:47:30.909926 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:30.909888    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xln8v" event={"ID":"6c498088-c01f-4ddd-9c52-0972d109f662","Type":"ContainerStarted","Data":"bc7e673a09c0ccdb4f4866faa2a046a566a75d3983a2d5b69cbc75387e6f4073"}
Jul 14 19:47:30.912356 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:30.912335    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal" event={"ID":"b9aaaac1f8a2716b7bd3ac221c06559b","Type":"ContainerStarted","Data":"fef4620dd1e367edec7a535de07ba0d9184ef87692b5d41f795a72ff3728cea6"}
Jul 14 19:47:30.913954 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:30.913922    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kmfht" event={"ID":"a52eed96-0a8b-4a11-99d1-7169c12bfa28","Type":"ContainerStarted","Data":"5e4e3b8312c17d74276c788167fd973f9d32eefb3e188e82ea69a0dd3aea7488"}
Jul 14 19:47:30.915380 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:30.915275    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6" event={"ID":"4a36af4c-9eac-465f-9a0e-d65a382530f7","Type":"ContainerStarted","Data":"4e35dc16909c141ecf0b1f63ab2873003411960f8b2bbeb608d2aae8da3b80a6"}
Jul 14 19:47:30.917062 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:30.917002    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qwdn7" event={"ID":"1b5468c6-fd78-4acc-b882-cf389722a101","Type":"ContainerStarted","Data":"b8aca118c8492d56e4c3cbafab5f9f9eb70c16e901641d2cd9e639db9927c322"}
Jul 14 19:47:30.918405 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:30.918348    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-smrzh" event={"ID":"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937","Type":"ContainerStarted","Data":"45490f85902a150c2c97ac804a6b15d498072b5861f0ecd3f22f83331f4daeb9"}
Jul 14 19:47:31.393661 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:31.393319    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/108c0b81-209c-46de-8d27-aba218077646-metrics-certs\") pod \"network-metrics-daemon-z2mcj\" (UID: \"108c0b81-209c-46de-8d27-aba218077646\") " pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:47:31.393826 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:31.393581    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 19:47:31.393826 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:31.393783    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/108c0b81-209c-46de-8d27-aba218077646-metrics-certs podName:108c0b81-209c-46de-8d27-aba218077646 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:33.393760852 +0000 UTC m=+5.994064533 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/108c0b81-209c-46de-8d27-aba218077646-metrics-certs") pod "network-metrics-daemon-z2mcj" (UID: "108c0b81-209c-46de-8d27-aba218077646") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 19:47:31.495108 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:31.494599    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/120711d8-0534-470b-ab0f-5433b06fef47-original-pull-secret\") pod \"global-pull-secret-syncer-pp59q\" (UID: \"120711d8-0534-470b-ab0f-5433b06fef47\") " pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:31.495108 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:31.494744    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 19:47:31.495108 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:31.494800    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/120711d8-0534-470b-ab0f-5433b06fef47-original-pull-secret podName:120711d8-0534-470b-ab0f-5433b06fef47 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:33.4947835 +0000 UTC m=+6.095087187 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/120711d8-0534-470b-ab0f-5433b06fef47-original-pull-secret") pod "global-pull-secret-syncer-pp59q" (UID: "120711d8-0534-470b-ab0f-5433b06fef47") : object "kube-system"/"original-pull-secret" not registered
Jul 14 19:47:31.595759 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:31.595724    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncmfq\" (UniqueName: \"kubernetes.io/projected/a11b1aab-78a3-48da-904c-71226e9c85ff-kube-api-access-ncmfq\") pod \"network-check-target-z785n\" (UID: \"a11b1aab-78a3-48da-904c-71226e9c85ff\") " pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:47:31.595982 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:31.595954    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 19:47:31.596059 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:31.595992    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 19:47:31.596059 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:31.596007    2560 projected.go:196] Error preparing data for projected volume kube-api-access-ncmfq for pod openshift-network-diagnostics/network-check-target-z785n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 19:47:31.596166 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:31.596060    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a11b1aab-78a3-48da-904c-71226e9c85ff-kube-api-access-ncmfq podName:a11b1aab-78a3-48da-904c-71226e9c85ff nodeName:}" failed. No retries permitted until 2026-07-14 19:47:33.596041571 +0000 UTC m=+6.196345256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ncmfq" (UniqueName: "kubernetes.io/projected/a11b1aab-78a3-48da-904c-71226e9c85ff-kube-api-access-ncmfq") pod "network-check-target-z785n" (UID: "a11b1aab-78a3-48da-904c-71226e9c85ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 19:47:31.817722 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:31.817638    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 19:42:29 +0000 UTC" deadline="2028-04-24 16:20:30.790783054 +0000 UTC"
Jul 14 19:47:31.817722 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:31.817677    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15596h32m58.973110334s"
Jul 14 19:47:31.885562 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:31.885307    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:31.885562 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:31.885435    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp59q" podUID="120711d8-0534-470b-ab0f-5433b06fef47"
Jul 14 19:47:32.884628 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:32.883847    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:47:32.884628 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:32.883999    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z785n" podUID="a11b1aab-78a3-48da-904c-71226e9c85ff"
Jul 14 19:47:32.884628 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:32.884445    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:47:32.884628 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:32.884557    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2mcj" podUID="108c0b81-209c-46de-8d27-aba218077646"
Jul 14 19:47:33.410815 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:33.410769    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/108c0b81-209c-46de-8d27-aba218077646-metrics-certs\") pod \"network-metrics-daemon-z2mcj\" (UID: \"108c0b81-209c-46de-8d27-aba218077646\") " pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:47:33.410966 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:33.410946    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 19:47:33.411039 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:33.411009    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/108c0b81-209c-46de-8d27-aba218077646-metrics-certs podName:108c0b81-209c-46de-8d27-aba218077646 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:37.410990405 +0000 UTC m=+10.011294089 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/108c0b81-209c-46de-8d27-aba218077646-metrics-certs") pod "network-metrics-daemon-z2mcj" (UID: "108c0b81-209c-46de-8d27-aba218077646") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 19:47:33.512254 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:33.512212    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/120711d8-0534-470b-ab0f-5433b06fef47-original-pull-secret\") pod \"global-pull-secret-syncer-pp59q\" (UID: \"120711d8-0534-470b-ab0f-5433b06fef47\") " pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:33.512428 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:33.512380    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 19:47:33.512482 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:33.512441    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/120711d8-0534-470b-ab0f-5433b06fef47-original-pull-secret podName:120711d8-0534-470b-ab0f-5433b06fef47 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:37.512422601 +0000 UTC m=+10.112726287 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/120711d8-0534-470b-ab0f-5433b06fef47-original-pull-secret") pod "global-pull-secret-syncer-pp59q" (UID: "120711d8-0534-470b-ab0f-5433b06fef47") : object "kube-system"/"original-pull-secret" not registered
Jul 14 19:47:33.613547 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:33.613506    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncmfq\" (UniqueName: \"kubernetes.io/projected/a11b1aab-78a3-48da-904c-71226e9c85ff-kube-api-access-ncmfq\") pod \"network-check-target-z785n\" (UID: \"a11b1aab-78a3-48da-904c-71226e9c85ff\") " pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:47:33.623999 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:33.619457    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 19:47:33.623999 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:33.619491    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 19:47:33.623999 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:33.619516    2560 projected.go:196] Error preparing data for projected volume kube-api-access-ncmfq for pod openshift-network-diagnostics/network-check-target-z785n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 19:47:33.623999 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:33.619591    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a11b1aab-78a3-48da-904c-71226e9c85ff-kube-api-access-ncmfq podName:a11b1aab-78a3-48da-904c-71226e9c85ff nodeName:}" failed. No retries permitted until 2026-07-14 19:47:37.619561176 +0000 UTC m=+10.219864873 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ncmfq" (UniqueName: "kubernetes.io/projected/a11b1aab-78a3-48da-904c-71226e9c85ff-kube-api-access-ncmfq") pod "network-check-target-z785n" (UID: "a11b1aab-78a3-48da-904c-71226e9c85ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 19:47:33.886219 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:33.885663    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:33.886219 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:33.885809    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp59q" podUID="120711d8-0534-470b-ab0f-5433b06fef47"
Jul 14 19:47:34.884711 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:34.884016    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:47:34.884711 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:34.884137    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z785n" podUID="a11b1aab-78a3-48da-904c-71226e9c85ff"
Jul 14 19:47:34.884711 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:34.884523    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:47:34.884711 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:34.884626    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2mcj" podUID="108c0b81-209c-46de-8d27-aba218077646"
Jul 14 19:47:35.886708 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:35.886674    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:35.887102 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:35.886809    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp59q" podUID="120711d8-0534-470b-ab0f-5433b06fef47"
Jul 14 19:47:36.883484 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:36.883449    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:47:36.883686 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:36.883456    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:47:36.883686 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:36.883594    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z785n" podUID="a11b1aab-78a3-48da-904c-71226e9c85ff"
Jul 14 19:47:36.883686 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:36.883672    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2mcj" podUID="108c0b81-209c-46de-8d27-aba218077646"
Jul 14 19:47:37.445137 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:37.445106    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/108c0b81-209c-46de-8d27-aba218077646-metrics-certs\") pod \"network-metrics-daemon-z2mcj\" (UID: \"108c0b81-209c-46de-8d27-aba218077646\") " pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:47:37.445554 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:37.445315    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 19:47:37.445554 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:37.445385    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/108c0b81-209c-46de-8d27-aba218077646-metrics-certs podName:108c0b81-209c-46de-8d27-aba218077646 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:45.445369224 +0000 UTC m=+18.045672930 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/108c0b81-209c-46de-8d27-aba218077646-metrics-certs") pod "network-metrics-daemon-z2mcj" (UID: "108c0b81-209c-46de-8d27-aba218077646") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 19:47:37.545873 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:37.545838    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/120711d8-0534-470b-ab0f-5433b06fef47-original-pull-secret\") pod \"global-pull-secret-syncer-pp59q\" (UID: \"120711d8-0534-470b-ab0f-5433b06fef47\") " pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:37.546031 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:37.545970    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 19:47:37.546031 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:37.546024    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/120711d8-0534-470b-ab0f-5433b06fef47-original-pull-secret podName:120711d8-0534-470b-ab0f-5433b06fef47 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:45.546011732 +0000 UTC m=+18.146315413 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/120711d8-0534-470b-ab0f-5433b06fef47-original-pull-secret") pod "global-pull-secret-syncer-pp59q" (UID: "120711d8-0534-470b-ab0f-5433b06fef47") : object "kube-system"/"original-pull-secret" not registered
Jul 14 19:47:37.646989 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:37.646951    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncmfq\" (UniqueName: \"kubernetes.io/projected/a11b1aab-78a3-48da-904c-71226e9c85ff-kube-api-access-ncmfq\") pod \"network-check-target-z785n\" (UID: \"a11b1aab-78a3-48da-904c-71226e9c85ff\") " pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:47:37.647166 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:37.647145    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 19:47:37.647291 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:37.647189    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 19:47:37.647291 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:37.647202    2560 projected.go:196] Error preparing data for projected volume kube-api-access-ncmfq for pod openshift-network-diagnostics/network-check-target-z785n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 19:47:37.647291 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:37.647259    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a11b1aab-78a3-48da-904c-71226e9c85ff-kube-api-access-ncmfq podName:a11b1aab-78a3-48da-904c-71226e9c85ff nodeName:}" failed. No retries permitted until 2026-07-14 19:47:45.647239994 +0000 UTC m=+18.247543682 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ncmfq" (UniqueName: "kubernetes.io/projected/a11b1aab-78a3-48da-904c-71226e9c85ff-kube-api-access-ncmfq") pod "network-check-target-z785n" (UID: "a11b1aab-78a3-48da-904c-71226e9c85ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 19:47:37.885336 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:37.885247    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:37.885486 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:37.885367    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp59q" podUID="120711d8-0534-470b-ab0f-5433b06fef47"
Jul 14 19:47:38.884429 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:38.884388    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:47:38.884891 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:38.884503    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z785n" podUID="a11b1aab-78a3-48da-904c-71226e9c85ff"
Jul 14 19:47:38.884967 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:38.884922    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:47:38.885102 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:38.885025    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2mcj" podUID="108c0b81-209c-46de-8d27-aba218077646"
Jul 14 19:47:39.884643 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:39.884206    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:39.884643 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:39.884342    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp59q" podUID="120711d8-0534-470b-ab0f-5433b06fef47"
Jul 14 19:47:40.883822 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:40.883786    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:47:40.883985 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:40.883839    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:47:40.883985 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:40.883933    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2mcj" podUID="108c0b81-209c-46de-8d27-aba218077646"
Jul 14 19:47:40.884114 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:40.884062    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z785n" podUID="a11b1aab-78a3-48da-904c-71226e9c85ff"
Jul 14 19:47:41.884189 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:41.884146    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:41.884533 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:41.884277    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp59q" podUID="120711d8-0534-470b-ab0f-5433b06fef47"
Jul 14 19:47:42.883627 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:42.883586    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:47:42.883627 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:42.883634    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:47:42.883864 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:42.883724    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2mcj" podUID="108c0b81-209c-46de-8d27-aba218077646"
Jul 14 19:47:42.883923 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:42.883854    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z785n" podUID="a11b1aab-78a3-48da-904c-71226e9c85ff"
Jul 14 19:47:43.883949 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:43.883914    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:43.884447 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:43.884038    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp59q" podUID="120711d8-0534-470b-ab0f-5433b06fef47"
Jul 14 19:47:44.883738 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:44.883703    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:47:44.883969 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:44.883836    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z785n" podUID="a11b1aab-78a3-48da-904c-71226e9c85ff"
Jul 14 19:47:44.883969 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:44.883900    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:47:44.884408 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:44.884026    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2mcj" podUID="108c0b81-209c-46de-8d27-aba218077646"
Jul 14 19:47:45.507187 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:45.507127    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/108c0b81-209c-46de-8d27-aba218077646-metrics-certs\") pod \"network-metrics-daemon-z2mcj\" (UID: \"108c0b81-209c-46de-8d27-aba218077646\") " pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:47:45.507476 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:45.507300    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 19:47:45.507476 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:45.507371    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/108c0b81-209c-46de-8d27-aba218077646-metrics-certs podName:108c0b81-209c-46de-8d27-aba218077646 nodeName:}" failed. No retries permitted until 2026-07-14 19:48:01.507350808 +0000 UTC m=+34.107654490 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/108c0b81-209c-46de-8d27-aba218077646-metrics-certs") pod "network-metrics-daemon-z2mcj" (UID: "108c0b81-209c-46de-8d27-aba218077646") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 19:47:45.607572 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:45.607526    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/120711d8-0534-470b-ab0f-5433b06fef47-original-pull-secret\") pod \"global-pull-secret-syncer-pp59q\" (UID: \"120711d8-0534-470b-ab0f-5433b06fef47\") " pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:45.607749 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:45.607681    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 19:47:45.607806 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:45.607751    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/120711d8-0534-470b-ab0f-5433b06fef47-original-pull-secret podName:120711d8-0534-470b-ab0f-5433b06fef47 nodeName:}" failed. No retries permitted until 2026-07-14 19:48:01.607735214 +0000 UTC m=+34.208038895 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/120711d8-0534-470b-ab0f-5433b06fef47-original-pull-secret") pod "global-pull-secret-syncer-pp59q" (UID: "120711d8-0534-470b-ab0f-5433b06fef47") : object "kube-system"/"original-pull-secret" not registered
Jul 14 19:47:45.707922 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:45.707880    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncmfq\" (UniqueName: \"kubernetes.io/projected/a11b1aab-78a3-48da-904c-71226e9c85ff-kube-api-access-ncmfq\") pod \"network-check-target-z785n\" (UID: \"a11b1aab-78a3-48da-904c-71226e9c85ff\") " pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:47:45.708105 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:45.708079    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 19:47:45.708105 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:45.708099    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 19:47:45.708105 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:45.708110    2560 projected.go:196] Error preparing data for projected volume kube-api-access-ncmfq for pod openshift-network-diagnostics/network-check-target-z785n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 19:47:45.708368 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:45.708185    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a11b1aab-78a3-48da-904c-71226e9c85ff-kube-api-access-ncmfq podName:a11b1aab-78a3-48da-904c-71226e9c85ff nodeName:}" failed. No retries permitted until 2026-07-14 19:48:01.708151144 +0000 UTC m=+34.308454830 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ncmfq" (UniqueName: "kubernetes.io/projected/a11b1aab-78a3-48da-904c-71226e9c85ff-kube-api-access-ncmfq") pod "network-check-target-z785n" (UID: "a11b1aab-78a3-48da-904c-71226e9c85ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 19:47:45.884384 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:45.884347    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:45.884817 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:45.884491    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp59q" podUID="120711d8-0534-470b-ab0f-5433b06fef47"
Jul 14 19:47:46.883598 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:46.883566    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:47:46.883780 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:46.883607    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:47:46.883780 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:46.883688    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z785n" podUID="a11b1aab-78a3-48da-904c-71226e9c85ff"
Jul 14 19:47:46.883780 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:46.883749    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2mcj" podUID="108c0b81-209c-46de-8d27-aba218077646"
Jul 14 19:47:47.884218 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:47.884186    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:47.884680 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:47.884285    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp59q" podUID="120711d8-0534-470b-ab0f-5433b06fef47"
Jul 14 19:47:48.884147 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:48.884103    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:47:48.884147 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:48.884143    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:47:48.884700 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:48.884250    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z785n" podUID="a11b1aab-78a3-48da-904c-71226e9c85ff"
Jul 14 19:47:48.884700 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:48.884384    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2mcj" podUID="108c0b81-209c-46de-8d27-aba218077646"
Jul 14 19:47:49.883462 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:49.883419    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:49.883744 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:49.883542    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp59q" podUID="120711d8-0534-470b-ab0f-5433b06fef47"
Jul 14 19:47:50.884139 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:50.883853    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:47:50.884832 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:50.883917    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:47:50.884832 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:50.884257    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z785n" podUID="a11b1aab-78a3-48da-904c-71226e9c85ff"
Jul 14 19:47:50.884832 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:50.884289    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2mcj" podUID="108c0b81-209c-46de-8d27-aba218077646"
Jul 14 19:47:50.964752 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:50.964660    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bz7rb" event={"ID":"ced44ef1-063d-48b9-81bc-412dd2a95f74","Type":"ContainerStarted","Data":"a0491bbff004dbac45bac7f9bfac2a40588c1ea79d11f8ce3d9e67623e2bf127"}
Jul 14 19:47:50.966260 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:50.966231    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9mcfm" event={"ID":"a305099d-5c23-4bad-bd9f-85c0461aa3ef","Type":"ContainerStarted","Data":"02492b7f7e67041bf1fce5841a15496cceb697d732f965827ab0b6913cfa9d53"}
Jul 14 19:47:50.967669 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:50.967645    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw" event={"ID":"d979e995-e1cd-4e85-bf27-dd797c072293","Type":"ContainerStarted","Data":"7d1563bee65546a795418d667602a0f0bded38e7dbdf308b97d90f166ae3f886"}
Jul 14 19:47:50.969252 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:50.969229    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xln8v" event={"ID":"6c498088-c01f-4ddd-9c52-0972d109f662","Type":"ContainerStarted","Data":"f57a8e7b2834a78361dae217c8b77e1f62854c6c1a392be7a78f349e3596f66c"}
Jul 14 19:47:50.970806 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:50.970782    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-168.ec2.internal" event={"ID":"0536b894286084f9faaec29be192da95","Type":"ContainerStarted","Data":"b02cb628867571d576270945c5809cdfcf24788e51e8e10736cbf2dbc3e9912f"}
Jul 14 19:47:50.972431 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:50.972409    2560 generic.go:356] "Generic (PLEG): container finished" podID="b9aaaac1f8a2716b7bd3ac221c06559b" containerID="794641e59db7fc1730d7bfef5e5b1ef1eeb44024e672b5bf1358715004d170da" exitCode=0
Jul 14 19:47:50.972528 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:50.972498    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal" event={"ID":"b9aaaac1f8a2716b7bd3ac221c06559b","Type":"ContainerDied","Data":"794641e59db7fc1730d7bfef5e5b1ef1eeb44024e672b5bf1358715004d170da"}
Jul 14 19:47:50.974220 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:50.974200    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kmfht" event={"ID":"a52eed96-0a8b-4a11-99d1-7169c12bfa28","Type":"ContainerStarted","Data":"12d0d609bb1ed6aae75593884f0b1ea3d2694b2d6f8bab0222e6f0f1a0fd3c04"}
Jul 14 19:47:50.977063 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:50.977048    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tzr6_4a36af4c-9eac-465f-9a0e-d65a382530f7/ovn-acl-logging/0.log"
Jul 14 19:47:50.977389 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:50.977372    2560 generic.go:356] "Generic (PLEG): container finished" podID="4a36af4c-9eac-465f-9a0e-d65a382530f7" containerID="447ec6629382cac414170c79dd0036ea59eba9d1b5caa60cd268336849d3bd6c" exitCode=1
Jul 14 19:47:50.977451 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:50.977428    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6" event={"ID":"4a36af4c-9eac-465f-9a0e-d65a382530f7","Type":"ContainerStarted","Data":"7b88fa6be415332f127f725bd36204fcf1572bc1c9dc072f6086974879557964"}
Jul 14 19:47:50.977505 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:50.977480    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6" event={"ID":"4a36af4c-9eac-465f-9a0e-d65a382530f7","Type":"ContainerStarted","Data":"dba564a222174ba2ecdb64f7445fb60a5abffc3153d995cf4ad79645016085db"}
Jul 14 19:47:50.977552 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:50.977512    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6" event={"ID":"4a36af4c-9eac-465f-9a0e-d65a382530f7","Type":"ContainerStarted","Data":"9174ea44f2303951f003706e1ea639d0f920f7675cc4ec94b152a538e91d20fe"}
Jul 14 19:47:50.977589 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:50.977551    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6" event={"ID":"4a36af4c-9eac-465f-9a0e-d65a382530f7","Type":"ContainerStarted","Data":"a59d93ed64f7d2f954e854707d3989654e059abf828d81ca6db51d6ac1722264"}
Jul 14 19:47:50.977634 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:50.977589    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6" event={"ID":"4a36af4c-9eac-465f-9a0e-d65a382530f7","Type":"ContainerDied","Data":"447ec6629382cac414170c79dd0036ea59eba9d1b5caa60cd268336849d3bd6c"}
Jul 14 19:47:50.977634 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:50.977631    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6" event={"ID":"4a36af4c-9eac-465f-9a0e-d65a382530f7","Type":"ContainerStarted","Data":"609c8a1faec116f5452787d5f9e3deef1a31417e30693ce9016e217a17f82efb"}
Jul 14 19:47:50.978622 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:50.978603    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qwdn7" event={"ID":"1b5468c6-fd78-4acc-b882-cf389722a101","Type":"ContainerStarted","Data":"be08a17a402a36a8e3ef574da924d1b74d91a34fd44dc5b1b6fb95deaf7ab6e7"}
Jul 14 19:47:50.983383 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:50.981702    2560 generic.go:356] "Generic (PLEG): container finished" podID="b5bfcb6a-4113-4273-a3e2-d6b64f2e4937" containerID="d1fba0d765f9887ae09d15c538cd50524357a7da90143b0e0b1fd45be913dd7d" exitCode=0
Jul 14 19:47:50.983383 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:50.981736    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-smrzh" event={"ID":"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937","Type":"ContainerDied","Data":"d1fba0d765f9887ae09d15c538cd50524357a7da90143b0e0b1fd45be913dd7d"}
Jul 14 19:47:50.999087 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:50.999047    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9mcfm" podStartSLOduration=2.913189439 podStartE2EDuration="22.999034966s" totalImagesPullingTime="20.085845527s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:50.998797625 +0000 UTC m=+23.599101328" watchObservedRunningTime="2026-07-14 19:47:50.999034966 +0000 UTC m=+23.599338668"
Jul 14 19:47:50.999496 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:50.999469    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bz7rb" podStartSLOduration=2.958036899 podStartE2EDuration="22.999461221s" totalImagesPullingTime="20.041424322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:50.98244106 +0000 UTC m=+23.582744764" watchObservedRunningTime="2026-07-14 19:47:50.999461221 +0000 UTC m=+23.599764927"
Jul 14 19:47:51.029296 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:51.029253    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-168.ec2.internal" podStartSLOduration=22.029239312 podStartE2EDuration="22.029239312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:51.029013271 +0000 UTC m=+23.629316977" watchObservedRunningTime="2026-07-14 19:47:51.029239312 +0000 UTC m=+23.629543014"
Jul 14 19:47:51.065347 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:51.065296    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qwdn7" podStartSLOduration=3.012084543 podStartE2EDuration="23.065276161s" totalImagesPullingTime="20.053191618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:51.064791477 +0000 UTC m=+23.665095181" watchObservedRunningTime="2026-07-14 19:47:51.065276161 +0000 UTC m=+23.665579864"
Jul 14 19:47:51.082490 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:51.082442    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kmfht" podStartSLOduration=2.972981828 podStartE2EDuration="23.08241491s" totalImagesPullingTime="20.109433082s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:51.082222468 +0000 UTC m=+23.682526170" watchObservedRunningTime="2026-07-14 19:47:51.08241491 +0000 UTC m=+23.682718615"
Jul 14 19:47:51.599595 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:51.599560    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xln8v"
Jul 14 19:47:51.600143 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:51.600124    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xln8v"
Jul 14 19:47:51.614567 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:51.614524    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xln8v" podStartSLOduration=4.524780047 podStartE2EDuration="24.614510827s" totalImagesPullingTime="20.08973078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:51.101086693 +0000 UTC m=+23.701390395" watchObservedRunningTime="2026-07-14 19:47:51.614510827 +0000 UTC m=+24.214814529"
Jul 14 19:47:51.859071 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:51.859045    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 14 19:47:51.884126 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:51.884102    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:51.884274 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:51.884204    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp59q" podUID="120711d8-0534-470b-ab0f-5433b06fef47"
Jul 14 19:47:51.984959 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:51.984919    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal" event={"ID":"b9aaaac1f8a2716b7bd3ac221c06559b","Type":"ContainerStarted","Data":"c2aa50a8537f12fec4b4de6b2fd5525f257211a1339d2986238422cf0e07fe57"}
Jul 14 19:47:51.986395 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:51.986368    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rkcxf" event={"ID":"43a2ff43-5f27-4ad1-8349-28d398a95a63","Type":"ContainerStarted","Data":"7dd4aa7b0a8d562c265f773c49953ed5568aea9581c4e56ab0af458327bceba2"}
Jul 14 19:47:51.988055 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:51.988025    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw" event={"ID":"d979e995-e1cd-4e85-bf27-dd797c072293","Type":"ContainerStarted","Data":"a4eb045dede677aa2adc19b1fc806478a6a935624569a1aedb1c9843b17ffd62"}
Jul 14 19:47:51.988836 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:51.988811    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xln8v"
Jul 14 19:47:51.989267 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:51.989244    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xln8v"
Jul 14 19:47:51.999467 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:51.999431    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal" podStartSLOduration=22.999419235 podStartE2EDuration="22.999419235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:51.998989419 +0000 UTC m=+24.599293133" watchObservedRunningTime="2026-07-14 19:47:51.999419235 +0000 UTC m=+24.599722938"
Jul 14 19:47:52.026021 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:52.025940    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rkcxf" podStartSLOduration=12.287638198 podStartE2EDuration="24.025928439s" totalImagesPullingTime="11.738290241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:52.025656222 +0000 UTC m=+24.625959924" watchObservedRunningTime="2026-07-14 19:47:52.025928439 +0000 UTC m=+24.626232141"
Jul 14 19:47:52.840163 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:52.840058    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T19:47:51.859062072Z","UUID":"a72d35d2-3e6e-4b09-b3bc-936a80abce9e","Handler":null,"Name":"","Endpoint":""}
Jul 14 19:47:52.841673 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:52.841653    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 14 19:47:52.841802 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:52.841694    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 14 19:47:52.883600 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:52.883568    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:47:52.883600 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:52.883594    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:47:52.883822 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:52.883680    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z785n" podUID="a11b1aab-78a3-48da-904c-71226e9c85ff"
Jul 14 19:47:52.883822 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:52.883781    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2mcj" podUID="108c0b81-209c-46de-8d27-aba218077646"
Jul 14 19:47:53.884472 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:53.884289    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:53.884912 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:53.884584    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp59q" podUID="120711d8-0534-470b-ab0f-5433b06fef47"
Jul 14 19:47:53.996545 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:53.996514    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tzr6_4a36af4c-9eac-465f-9a0e-d65a382530f7/ovn-acl-logging/0.log"
Jul 14 19:47:53.996965 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:53.996938    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6" event={"ID":"4a36af4c-9eac-465f-9a0e-d65a382530f7","Type":"ContainerStarted","Data":"0ece9175dd694a5baa2946882ebdfa59e58ddaaee21267624d2b98c5c4d86911"}
Jul 14 19:47:54.883585 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:54.883557    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:47:54.883585 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:54.883582    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:47:54.883839 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:54.883666    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z785n" podUID="a11b1aab-78a3-48da-904c-71226e9c85ff"
Jul 14 19:47:54.883839 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:54.883830    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2mcj" podUID="108c0b81-209c-46de-8d27-aba218077646"
Jul 14 19:47:55.002066 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:55.002027    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw" event={"ID":"d979e995-e1cd-4e85-bf27-dd797c072293","Type":"ContainerStarted","Data":"1bdb5865b9975e31e861bfd67c5d5618aee4fab22b044184da162cde056fcbda"}
Jul 14 19:47:55.020473 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:55.020421    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nhtw" podStartSLOduration=12.770702743 podStartE2EDuration="28.020404008s" totalImagesPullingTime="15.249701265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:47:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:55.020245557 +0000 UTC m=+27.620549263" watchObservedRunningTime="2026-07-14 19:47:55.020404008 +0000 UTC m=+27.620707711"
Jul 14 19:47:55.883918 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:55.883609    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:55.884076 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:55.883995    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp59q" podUID="120711d8-0534-470b-ab0f-5433b06fef47"
Jul 14 19:47:56.008882 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:56.008854    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tzr6_4a36af4c-9eac-465f-9a0e-d65a382530f7/ovn-acl-logging/0.log"
Jul 14 19:47:56.009663 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:56.009320    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6" event={"ID":"4a36af4c-9eac-465f-9a0e-d65a382530f7","Type":"ContainerStarted","Data":"0aea5ad7fd3fa3b7329114ed25bbb8f2ab3b07088c14b1d1d41135e4f52cd6a0"}
Jul 14 19:47:56.009858 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:56.009842    2560 scope.go:117] "RemoveContainer" containerID="447ec6629382cac414170c79dd0036ea59eba9d1b5caa60cd268336849d3bd6c"
Jul 14 19:47:56.883689 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:56.883652    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:47:56.883863 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:56.883651    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:47:56.883863 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:56.883788    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2mcj" podUID="108c0b81-209c-46de-8d27-aba218077646"
Jul 14 19:47:56.883989 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:56.883864    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z785n" podUID="a11b1aab-78a3-48da-904c-71226e9c85ff"
Jul 14 19:47:57.884384 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:57.884350    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:57.884816 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:57.884434    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp59q" podUID="120711d8-0534-470b-ab0f-5433b06fef47"
Jul 14 19:47:58.015097 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:58.015063    2560 generic.go:356] "Generic (PLEG): container finished" podID="b5bfcb6a-4113-4273-a3e2-d6b64f2e4937" containerID="8e1bb89cce3e13923b8ecc49fffc1b1ff6adbb85d9015bc37af4bcfa1b216952" exitCode=0
Jul 14 19:47:58.015288 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:58.015142    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-smrzh" event={"ID":"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937","Type":"ContainerDied","Data":"8e1bb89cce3e13923b8ecc49fffc1b1ff6adbb85d9015bc37af4bcfa1b216952"}
Jul 14 19:47:58.019065 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:58.019046    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tzr6_4a36af4c-9eac-465f-9a0e-d65a382530f7/ovn-acl-logging/0.log"
Jul 14 19:47:58.019476 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:58.019449    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6" event={"ID":"4a36af4c-9eac-465f-9a0e-d65a382530f7","Type":"ContainerStarted","Data":"a7e31a4a6a1327c820dd22a7540dda4ee7d1693305f8f1c33410c2903ce5cc21"}
Jul 14 19:47:58.019717 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:58.019701    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:58.019804 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:58.019769    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:58.019859 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:58.019813    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:58.035228 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:58.035202    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:58.035461 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:58.035446    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:47:58.069924 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:58.068268    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6" podStartSLOduration=9.950583164 podStartE2EDuration="30.068244215s" totalImagesPullingTime="20.117661051s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:58.066045345 +0000 UTC m=+30.666349038" watchObservedRunningTime="2026-07-14 19:47:58.068244215 +0000 UTC m=+30.668547921"
Jul 14 19:47:58.883636 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:58.883605    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:47:58.883809 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:58.883742    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z785n" podUID="a11b1aab-78a3-48da-904c-71226e9c85ff"
Jul 14 19:47:58.883809 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:58.883785    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:47:58.883938 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:58.883919    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2mcj" podUID="108c0b81-209c-46de-8d27-aba218077646"
Jul 14 19:47:59.884089 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:47:59.884052    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:47:59.884459 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:47:59.884193    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp59q" podUID="120711d8-0534-470b-ab0f-5433b06fef47"
Jul 14 19:48:00.025286 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:00.025254    2560 generic.go:356] "Generic (PLEG): container finished" podID="b5bfcb6a-4113-4273-a3e2-d6b64f2e4937" containerID="1d8e44a3c37d0af337cec4bab8d631bdbad674bcd9c8ee957e3d99c4bf8a17a8" exitCode=0
Jul 14 19:48:00.025465 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:00.025346    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-smrzh" event={"ID":"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937","Type":"ContainerDied","Data":"1d8e44a3c37d0af337cec4bab8d631bdbad674bcd9c8ee957e3d99c4bf8a17a8"}
Jul 14 19:48:00.036191 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:00.036140    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6" podUID="4a36af4c-9eac-465f-9a0e-d65a382530f7" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 14 19:48:00.884237 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:00.884198    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:48:00.884237 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:00.884233    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:48:00.884643 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:00.884329    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z785n" podUID="a11b1aab-78a3-48da-904c-71226e9c85ff"
Jul 14 19:48:00.884643 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:00.884473    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2mcj" podUID="108c0b81-209c-46de-8d27-aba218077646"
Jul 14 19:48:01.029532 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:01.029504    2560 generic.go:356] "Generic (PLEG): container finished" podID="b5bfcb6a-4113-4273-a3e2-d6b64f2e4937" containerID="6a20de910673ae3da59307278fd10e0b071eadc7598adba161c6ace1aa8f9ed7" exitCode=0
Jul 14 19:48:01.029657 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:01.029544    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-smrzh" event={"ID":"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937","Type":"ContainerDied","Data":"6a20de910673ae3da59307278fd10e0b071eadc7598adba161c6ace1aa8f9ed7"}
Jul 14 19:48:01.523937 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:01.523906    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/108c0b81-209c-46de-8d27-aba218077646-metrics-certs\") pod \"network-metrics-daemon-z2mcj\" (UID: \"108c0b81-209c-46de-8d27-aba218077646\") " pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:48:01.524117 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:01.524034    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 19:48:01.524117 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:01.524085    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/108c0b81-209c-46de-8d27-aba218077646-metrics-certs podName:108c0b81-209c-46de-8d27-aba218077646 nodeName:}" failed. No retries permitted until 2026-07-14 19:48:33.524070753 +0000 UTC m=+66.124374434 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/108c0b81-209c-46de-8d27-aba218077646-metrics-certs") pod "network-metrics-daemon-z2mcj" (UID: "108c0b81-209c-46de-8d27-aba218077646") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 19:48:01.624503 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:01.624469    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/120711d8-0534-470b-ab0f-5433b06fef47-original-pull-secret\") pod \"global-pull-secret-syncer-pp59q\" (UID: \"120711d8-0534-470b-ab0f-5433b06fef47\") " pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:48:01.624678 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:01.624606    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 19:48:01.624678 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:01.624665    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/120711d8-0534-470b-ab0f-5433b06fef47-original-pull-secret podName:120711d8-0534-470b-ab0f-5433b06fef47 nodeName:}" failed. No retries permitted until 2026-07-14 19:48:33.624646955 +0000 UTC m=+66.224950636 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/120711d8-0534-470b-ab0f-5433b06fef47-original-pull-secret") pod "global-pull-secret-syncer-pp59q" (UID: "120711d8-0534-470b-ab0f-5433b06fef47") : object "kube-system"/"original-pull-secret" not registered
Jul 14 19:48:01.725409 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:01.725380    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncmfq\" (UniqueName: \"kubernetes.io/projected/a11b1aab-78a3-48da-904c-71226e9c85ff-kube-api-access-ncmfq\") pod \"network-check-target-z785n\" (UID: \"a11b1aab-78a3-48da-904c-71226e9c85ff\") " pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:48:01.725583 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:01.725555    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 19:48:01.725583 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:01.725576    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 19:48:01.725583 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:01.725585    2560 projected.go:196] Error preparing data for projected volume kube-api-access-ncmfq for pod openshift-network-diagnostics/network-check-target-z785n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 19:48:01.725735 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:01.725648    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a11b1aab-78a3-48da-904c-71226e9c85ff-kube-api-access-ncmfq podName:a11b1aab-78a3-48da-904c-71226e9c85ff nodeName:}" failed. No retries permitted until 2026-07-14 19:48:33.725627335 +0000 UTC m=+66.325931077 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ncmfq" (UniqueName: "kubernetes.io/projected/a11b1aab-78a3-48da-904c-71226e9c85ff-kube-api-access-ncmfq") pod "network-check-target-z785n" (UID: "a11b1aab-78a3-48da-904c-71226e9c85ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 19:48:01.884033 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:01.883998    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:48:01.884214 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:01.884126    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp59q" podUID="120711d8-0534-470b-ab0f-5433b06fef47"
Jul 14 19:48:02.884281 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:02.884247    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:48:02.884677 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:02.884247    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:48:02.884677 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:02.884347    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z785n" podUID="a11b1aab-78a3-48da-904c-71226e9c85ff"
Jul 14 19:48:02.884677 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:02.884427    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2mcj" podUID="108c0b81-209c-46de-8d27-aba218077646"
Jul 14 19:48:03.884273 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:03.884225    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:48:03.884544 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:03.884363    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp59q" podUID="120711d8-0534-470b-ab0f-5433b06fef47"
Jul 14 19:48:04.883657 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:04.883619    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:48:04.883837 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:04.883672    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:48:04.883837 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:04.883753    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2mcj" podUID="108c0b81-209c-46de-8d27-aba218077646"
Jul 14 19:48:04.883940 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:04.883889    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z785n" podUID="a11b1aab-78a3-48da-904c-71226e9c85ff"
Jul 14 19:48:05.883990 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:05.883799    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:48:05.884387 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:05.884080    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp59q" podUID="120711d8-0534-470b-ab0f-5433b06fef47"
Jul 14 19:48:06.044979 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:06.044947    2560 generic.go:356] "Generic (PLEG): container finished" podID="b5bfcb6a-4113-4273-a3e2-d6b64f2e4937" containerID="e3c72d1bdfe0211fa9eeda7630d2660e41354c50e9efbf5ebd6c94e388a5a656" exitCode=0
Jul 14 19:48:06.045141 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:06.045015    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-smrzh" event={"ID":"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937","Type":"ContainerDied","Data":"e3c72d1bdfe0211fa9eeda7630d2660e41354c50e9efbf5ebd6c94e388a5a656"}
Jul 14 19:48:06.883889 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:06.883855    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:48:06.884099 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:06.883969    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2mcj" podUID="108c0b81-209c-46de-8d27-aba218077646"
Jul 14 19:48:06.884099 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:06.884029    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:48:06.884496 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:06.884136    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z785n" podUID="a11b1aab-78a3-48da-904c-71226e9c85ff"
Jul 14 19:48:07.050198 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:07.050146    2560 generic.go:356] "Generic (PLEG): container finished" podID="b5bfcb6a-4113-4273-a3e2-d6b64f2e4937" containerID="a00344c25bd0a1e2b9c35e88c7f77be485e576f6a280c0315ccd716f895c171d" exitCode=0
Jul 14 19:48:07.050385 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:07.050212    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-smrzh" event={"ID":"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937","Type":"ContainerDied","Data":"a00344c25bd0a1e2b9c35e88c7f77be485e576f6a280c0315ccd716f895c171d"}
Jul 14 19:48:07.884594 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:07.884559    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:48:07.885096 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:07.884647    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp59q" podUID="120711d8-0534-470b-ab0f-5433b06fef47"
Jul 14 19:48:08.056905 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:08.056874    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-smrzh" event={"ID":"b5bfcb6a-4113-4273-a3e2-d6b64f2e4937","Type":"ContainerStarted","Data":"acdc3d544def9830c8cecbeabf409b1668d3649585d57653d63cec210bf70ad6"}
Jul 14 19:48:08.079569 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:08.079524    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-smrzh" podStartSLOduration=7.248733668 podStartE2EDuration="40.079508188s" totalImagesPullingTime="32.83077452s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:47:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:48:08.078385994 +0000 UTC m=+40.678689697" watchObservedRunningTime="2026-07-14 19:48:08.079508188 +0000 UTC m=+40.679811890"
Jul 14 19:48:08.508335 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:08.508297    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pp59q"]
Jul 14 19:48:08.508495 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:08.508401    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:48:08.508548 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:08.508504    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp59q" podUID="120711d8-0534-470b-ab0f-5433b06fef47"
Jul 14 19:48:08.511240 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:08.511208    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z785n"]
Jul 14 19:48:08.511391 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:08.511316    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:48:08.511461 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:08.511398    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z785n" podUID="a11b1aab-78a3-48da-904c-71226e9c85ff"
Jul 14 19:48:08.512034 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:08.512007    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z2mcj"]
Jul 14 19:48:08.512148 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:08.512121    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:48:08.512266 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:08.512243    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2mcj" podUID="108c0b81-209c-46de-8d27-aba218077646"
Jul 14 19:48:09.883960 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:09.883921    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:48:09.883960 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:09.883948    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:48:09.884703 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:09.883921    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:48:09.884703 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:09.884052    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp59q" podUID="120711d8-0534-470b-ab0f-5433b06fef47"
Jul 14 19:48:09.884703 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:09.884115    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z785n" podUID="a11b1aab-78a3-48da-904c-71226e9c85ff"
Jul 14 19:48:09.884703 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:09.884166    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2mcj" podUID="108c0b81-209c-46de-8d27-aba218077646"
Jul 14 19:48:11.883616 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:11.883426    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:48:11.884036 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:11.883453    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:48:11.884036 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:11.883705    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp59q" podUID="120711d8-0534-470b-ab0f-5433b06fef47"
Jul 14 19:48:11.884036 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:11.883454    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:48:11.884036 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:11.883784    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z785n" podUID="a11b1aab-78a3-48da-904c-71226e9c85ff"
Jul 14 19:48:11.884036 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:11.883906    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z2mcj" podUID="108c0b81-209c-46de-8d27-aba218077646"
Jul 14 19:48:12.291866 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.291838    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeReady"
Jul 14 19:48:12.292037 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.291975    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 14 19:48:12.344218 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.344167    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mgqdj"]
Jul 14 19:48:12.358189 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.358150    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-d2d6k"]
Jul 14 19:48:12.358309 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.358274    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mgqdj"
Jul 14 19:48:12.361214 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.361189    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-949mk\""
Jul 14 19:48:12.361382 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.361220    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 14 19:48:12.361508 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.361222    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 14 19:48:12.372314 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.372296    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d2d6k"]
Jul 14 19:48:12.372408 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.372337    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mgqdj"]
Jul 14 19:48:12.372408 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.372365    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fd6mw"]
Jul 14 19:48:12.372408 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.372393    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d2d6k"
Jul 14 19:48:12.376269 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.376026    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 14 19:48:12.376269 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.376043    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 14 19:48:12.376269 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.376074    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 14 19:48:12.376269 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.376102    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-stdzp\""
Jul 14 19:48:12.376512 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.376350    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 14 19:48:12.394249 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.394221    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fd6mw"]
Jul 14 19:48:12.394385 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.394317    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fd6mw"
Jul 14 19:48:12.396633 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.396606    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nfcd5\""
Jul 14 19:48:12.396734 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.396666    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 14 19:48:12.396843 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.396828    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 14 19:48:12.397047 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.397035    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 14 19:48:12.500526 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.500491    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bjqq\" (UniqueName: \"kubernetes.io/projected/8fafc082-f569-4875-8d12-db7504865e6d-kube-api-access-7bjqq\") pod \"dns-default-mgqdj\" (UID: \"8fafc082-f569-4875-8d12-db7504865e6d\") " pod="openshift-dns/dns-default-mgqdj"
Jul 14 19:48:12.500526 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.500529    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8fafc082-f569-4875-8d12-db7504865e6d-config-volume\") pod \"dns-default-mgqdj\" (UID: \"8fafc082-f569-4875-8d12-db7504865e6d\") " pod="openshift-dns/dns-default-mgqdj"
Jul 14 19:48:12.500726 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.500555    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1fd75dce-40e5-494c-92c4-eac34f1fc302-crio-socket\") pod \"insights-runtime-extractor-d2d6k\" (UID: \"1fd75dce-40e5-494c-92c4-eac34f1fc302\") " pod="openshift-insights/insights-runtime-extractor-d2d6k"
Jul 14 19:48:12.500726 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.500622    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8fafc082-f569-4875-8d12-db7504865e6d-metrics-tls\") pod \"dns-default-mgqdj\" (UID: \"8fafc082-f569-4875-8d12-db7504865e6d\") " pod="openshift-dns/dns-default-mgqdj"
Jul 14 19:48:12.500726 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.500717    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86c2f79a-dd6d-46b1-ac5c-e1d00a7a652b-cert\") pod \"ingress-canary-fd6mw\" (UID: \"86c2f79a-dd6d-46b1-ac5c-e1d00a7a652b\") " pod="openshift-ingress-canary/ingress-canary-fd6mw"
Jul 14 19:48:12.500814 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.500740    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fd75dce-40e5-494c-92c4-eac34f1fc302-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d2d6k\" (UID: \"1fd75dce-40e5-494c-92c4-eac34f1fc302\") " pod="openshift-insights/insights-runtime-extractor-d2d6k"
Jul 14 19:48:12.500814 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.500792    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fafc082-f569-4875-8d12-db7504865e6d-tmp-dir\") pod \"dns-default-mgqdj\" (UID: \"8fafc082-f569-4875-8d12-db7504865e6d\") " pod="openshift-dns/dns-default-mgqdj"
Jul 14 19:48:12.500875 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.500813    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1fd75dce-40e5-494c-92c4-eac34f1fc302-data-volume\") pod \"insights-runtime-extractor-d2d6k\" (UID: \"1fd75dce-40e5-494c-92c4-eac34f1fc302\") " pod="openshift-insights/insights-runtime-extractor-d2d6k"
Jul 14 19:48:12.500875 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.500852    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2lql\" (UniqueName: \"kubernetes.io/projected/86c2f79a-dd6d-46b1-ac5c-e1d00a7a652b-kube-api-access-s2lql\") pod \"ingress-canary-fd6mw\" (UID: \"86c2f79a-dd6d-46b1-ac5c-e1d00a7a652b\") " pod="openshift-ingress-canary/ingress-canary-fd6mw"
Jul 14 19:48:12.500875 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.500868    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfb79\" (UniqueName: \"kubernetes.io/projected/1fd75dce-40e5-494c-92c4-eac34f1fc302-kube-api-access-gfb79\") pod \"insights-runtime-extractor-d2d6k\" (UID: \"1fd75dce-40e5-494c-92c4-eac34f1fc302\") " pod="openshift-insights/insights-runtime-extractor-d2d6k"
Jul 14 19:48:12.500963 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.500884    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fd75dce-40e5-494c-92c4-eac34f1fc302-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2d6k\" (UID: \"1fd75dce-40e5-494c-92c4-eac34f1fc302\") " pod="openshift-insights/insights-runtime-extractor-d2d6k"
Jul 14 19:48:12.601856 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.601775    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86c2f79a-dd6d-46b1-ac5c-e1d00a7a652b-cert\") pod \"ingress-canary-fd6mw\" (UID: \"86c2f79a-dd6d-46b1-ac5c-e1d00a7a652b\") " pod="openshift-ingress-canary/ingress-canary-fd6mw"
Jul 14 19:48:12.601856 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.601808    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fd75dce-40e5-494c-92c4-eac34f1fc302-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d2d6k\" (UID: \"1fd75dce-40e5-494c-92c4-eac34f1fc302\") " pod="openshift-insights/insights-runtime-extractor-d2d6k"
Jul 14 19:48:12.601856 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.601842    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fafc082-f569-4875-8d12-db7504865e6d-tmp-dir\") pod \"dns-default-mgqdj\" (UID: \"8fafc082-f569-4875-8d12-db7504865e6d\") " pod="openshift-dns/dns-default-mgqdj"
Jul 14 19:48:12.602076 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.601954    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1fd75dce-40e5-494c-92c4-eac34f1fc302-data-volume\") pod \"insights-runtime-extractor-d2d6k\" (UID: \"1fd75dce-40e5-494c-92c4-eac34f1fc302\") " pod="openshift-insights/insights-runtime-extractor-d2d6k"
Jul 14 19:48:12.602076 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.601990    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2lql\" (UniqueName: \"kubernetes.io/projected/86c2f79a-dd6d-46b1-ac5c-e1d00a7a652b-kube-api-access-s2lql\") pod \"ingress-canary-fd6mw\" (UID: \"86c2f79a-dd6d-46b1-ac5c-e1d00a7a652b\") " pod="openshift-ingress-canary/ingress-canary-fd6mw"
Jul 14 19:48:12.602076 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.602016    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfb79\" (UniqueName: \"kubernetes.io/projected/1fd75dce-40e5-494c-92c4-eac34f1fc302-kube-api-access-gfb79\") pod \"insights-runtime-extractor-d2d6k\" (UID: \"1fd75dce-40e5-494c-92c4-eac34f1fc302\") " pod="openshift-insights/insights-runtime-extractor-d2d6k"
Jul 14 19:48:12.602076 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.602048    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fd75dce-40e5-494c-92c4-eac34f1fc302-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2d6k\" (UID: \"1fd75dce-40e5-494c-92c4-eac34f1fc302\") " pod="openshift-insights/insights-runtime-extractor-d2d6k"
Jul 14 19:48:12.602276 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.602210    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fafc082-f569-4875-8d12-db7504865e6d-tmp-dir\") pod \"dns-default-mgqdj\" (UID: \"8fafc082-f569-4875-8d12-db7504865e6d\") " pod="openshift-dns/dns-default-mgqdj"
Jul 14 19:48:12.602276 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.602231    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bjqq\" (UniqueName: \"kubernetes.io/projected/8fafc082-f569-4875-8d12-db7504865e6d-kube-api-access-7bjqq\") pod \"dns-default-mgqdj\" (UID: \"8fafc082-f569-4875-8d12-db7504865e6d\") " pod="openshift-dns/dns-default-mgqdj"
Jul 14 19:48:12.602387 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.602279    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1fd75dce-40e5-494c-92c4-eac34f1fc302-data-volume\") pod \"insights-runtime-extractor-d2d6k\" (UID: \"1fd75dce-40e5-494c-92c4-eac34f1fc302\") " pod="openshift-insights/insights-runtime-extractor-d2d6k"
Jul 14 19:48:12.602387 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.602279    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8fafc082-f569-4875-8d12-db7504865e6d-config-volume\") pod \"dns-default-mgqdj\" (UID: \"8fafc082-f569-4875-8d12-db7504865e6d\") " pod="openshift-dns/dns-default-mgqdj"
Jul 14 19:48:12.602387 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.602346    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1fd75dce-40e5-494c-92c4-eac34f1fc302-crio-socket\") pod \"insights-runtime-extractor-d2d6k\" (UID: \"1fd75dce-40e5-494c-92c4-eac34f1fc302\") " pod="openshift-insights/insights-runtime-extractor-d2d6k"
Jul 14 19:48:12.602540 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.602409    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8fafc082-f569-4875-8d12-db7504865e6d-metrics-tls\") pod \"dns-default-mgqdj\" (UID: \"8fafc082-f569-4875-8d12-db7504865e6d\") " pod="openshift-dns/dns-default-mgqdj"
Jul 14 19:48:12.602540 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.602499    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fd75dce-40e5-494c-92c4-eac34f1fc302-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d2d6k\" (UID: \"1fd75dce-40e5-494c-92c4-eac34f1fc302\") " pod="openshift-insights/insights-runtime-extractor-d2d6k"
Jul 14 19:48:12.602637 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.602605    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1fd75dce-40e5-494c-92c4-eac34f1fc302-crio-socket\") pod \"insights-runtime-extractor-d2d6k\" (UID: \"1fd75dce-40e5-494c-92c4-eac34f1fc302\") " pod="openshift-insights/insights-runtime-extractor-d2d6k"
Jul 14 19:48:12.602921 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.602900    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8fafc082-f569-4875-8d12-db7504865e6d-config-volume\") pod \"dns-default-mgqdj\" (UID: \"8fafc082-f569-4875-8d12-db7504865e6d\") " pod="openshift-dns/dns-default-mgqdj"
Jul 14 19:48:12.606050 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.606028    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8fafc082-f569-4875-8d12-db7504865e6d-metrics-tls\") pod \"dns-default-mgqdj\" (UID: \"8fafc082-f569-4875-8d12-db7504865e6d\") " pod="openshift-dns/dns-default-mgqdj"
Jul 14 19:48:12.606143 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.606068    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fd75dce-40e5-494c-92c4-eac34f1fc302-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2d6k\" (UID: \"1fd75dce-40e5-494c-92c4-eac34f1fc302\") " pod="openshift-insights/insights-runtime-extractor-d2d6k"
Jul 14 19:48:12.606215 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.606160    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86c2f79a-dd6d-46b1-ac5c-e1d00a7a652b-cert\") pod \"ingress-canary-fd6mw\" (UID: \"86c2f79a-dd6d-46b1-ac5c-e1d00a7a652b\") " pod="openshift-ingress-canary/ingress-canary-fd6mw"
Jul 14 19:48:12.611359 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.611334    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2lql\" (UniqueName: \"kubernetes.io/projected/86c2f79a-dd6d-46b1-ac5c-e1d00a7a652b-kube-api-access-s2lql\") pod \"ingress-canary-fd6mw\" (UID: \"86c2f79a-dd6d-46b1-ac5c-e1d00a7a652b\") " pod="openshift-ingress-canary/ingress-canary-fd6mw"
Jul 14 19:48:12.611468 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.611360    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bjqq\" (UniqueName: \"kubernetes.io/projected/8fafc082-f569-4875-8d12-db7504865e6d-kube-api-access-7bjqq\") pod \"dns-default-mgqdj\" (UID: \"8fafc082-f569-4875-8d12-db7504865e6d\") " pod="openshift-dns/dns-default-mgqdj"
Jul 14 19:48:12.611468 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.611336    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfb79\" (UniqueName: \"kubernetes.io/projected/1fd75dce-40e5-494c-92c4-eac34f1fc302-kube-api-access-gfb79\") pod \"insights-runtime-extractor-d2d6k\" (UID: \"1fd75dce-40e5-494c-92c4-eac34f1fc302\") " pod="openshift-insights/insights-runtime-extractor-d2d6k"
Jul 14 19:48:12.669217 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.669168    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mgqdj"
Jul 14 19:48:12.684331 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.684306    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d2d6k"
Jul 14 19:48:12.702202 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.702146    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fd6mw"
Jul 14 19:48:12.884844 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.884772    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mgqdj"]
Jul 14 19:48:12.888339 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.888311    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d2d6k"]
Jul 14 19:48:12.888524 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:48:12.888500    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fafc082_f569_4875_8d12_db7504865e6d.slice/crio-a58cf80fd392134d8aec4bf194948598922548754be38c33e03d16a94cf9d9b6 WatchSource:0}: Error finding container a58cf80fd392134d8aec4bf194948598922548754be38c33e03d16a94cf9d9b6: Status 404 returned error can't find the container with id a58cf80fd392134d8aec4bf194948598922548754be38c33e03d16a94cf9d9b6
Jul 14 19:48:12.890658 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:12.890639    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fd6mw"]
Jul 14 19:48:12.891646 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:48:12.891589    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd75dce_40e5_494c_92c4_eac34f1fc302.slice/crio-30a86670117062f6e4e6ddeb39791a60bb9afe01c1e165d4599ad4a9fa0def0a WatchSource:0}: Error finding container 30a86670117062f6e4e6ddeb39791a60bb9afe01c1e165d4599ad4a9fa0def0a: Status 404 returned error can't find the container with id 30a86670117062f6e4e6ddeb39791a60bb9afe01c1e165d4599ad4a9fa0def0a
Jul 14 19:48:12.893705 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:48:12.893687    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c2f79a_dd6d_46b1_ac5c_e1d00a7a652b.slice/crio-114dcbdb5eb76f6659bbd71dfc5c62fb6b31d3ad346e65099d8189f2293bbeb3 WatchSource:0}: Error finding container 114dcbdb5eb76f6659bbd71dfc5c62fb6b31d3ad346e65099d8189f2293bbeb3: Status 404 returned error can't find the container with id 114dcbdb5eb76f6659bbd71dfc5c62fb6b31d3ad346e65099d8189f2293bbeb3
Jul 14 19:48:13.069043 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:13.069012    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fd6mw" event={"ID":"86c2f79a-dd6d-46b1-ac5c-e1d00a7a652b","Type":"ContainerStarted","Data":"114dcbdb5eb76f6659bbd71dfc5c62fb6b31d3ad346e65099d8189f2293bbeb3"}
Jul 14 19:48:13.069978 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:13.069960    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d2d6k" event={"ID":"1fd75dce-40e5-494c-92c4-eac34f1fc302","Type":"ContainerStarted","Data":"30a86670117062f6e4e6ddeb39791a60bb9afe01c1e165d4599ad4a9fa0def0a"}
Jul 14 19:48:13.070974 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:13.070955    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mgqdj" event={"ID":"8fafc082-f569-4875-8d12-db7504865e6d","Type":"ContainerStarted","Data":"a58cf80fd392134d8aec4bf194948598922548754be38c33e03d16a94cf9d9b6"}
Jul 14 19:48:13.885341 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:13.885302    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:48:13.885777 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:13.885661    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:48:13.886433 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:13.885908    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:48:13.891207 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:13.890048    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b6fln\""
Jul 14 19:48:13.891207 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:13.890355    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 14 19:48:13.891207 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:13.890418    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 14 19:48:13.891207 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:13.890355    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 14 19:48:13.891207 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:13.890798    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-45mdm\""
Jul 14 19:48:13.891207 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:13.891008    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 14 19:48:16.272120 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.272080    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56d7c87f44-j5b8q"]
Jul 14 19:48:16.420010 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.419978    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56d7c87f44-j5b8q"]
Jul 14 19:48:16.420147 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.420088    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56d7c87f44-j5b8q"
Jul 14 19:48:16.423293 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.423271    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 14 19:48:16.423293 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.423303    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 14 19:48:16.423477 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.423392    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 14 19:48:16.423477 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.423417    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-j75k8\""
Jul 14 19:48:16.423477 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.423445    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 14 19:48:16.423684 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.423671    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 14 19:48:16.424243 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.424223    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 14 19:48:16.424375 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.424225    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 14 19:48:16.429130 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.429108    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 14 19:48:16.434134 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.434116    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp5ks\" (UniqueName: \"kubernetes.io/projected/4a7acfc7-7b64-49e9-9310-a251605d06ce-kube-api-access-dp5ks\") pod \"console-56d7c87f44-j5b8q\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") " pod="openshift-console/console-56d7c87f44-j5b8q"
Jul 14 19:48:16.434253 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.434201    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a7acfc7-7b64-49e9-9310-a251605d06ce-console-serving-cert\") pod \"console-56d7c87f44-j5b8q\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") " pod="openshift-console/console-56d7c87f44-j5b8q"
Jul 14 19:48:16.434253 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.434231    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a7acfc7-7b64-49e9-9310-a251605d06ce-console-oauth-config\") pod \"console-56d7c87f44-j5b8q\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") " pod="openshift-console/console-56d7c87f44-j5b8q"
Jul 14 19:48:16.434362 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.434260    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a7acfc7-7b64-49e9-9310-a251605d06ce-trusted-ca-bundle\") pod \"console-56d7c87f44-j5b8q\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") " pod="openshift-console/console-56d7c87f44-j5b8q"
Jul 14 19:48:16.434362 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.434322    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a7acfc7-7b64-49e9-9310-a251605d06ce-console-config\") pod \"console-56d7c87f44-j5b8q\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") " pod="openshift-console/console-56d7c87f44-j5b8q"
Jul 14 19:48:16.434362 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.434347    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a7acfc7-7b64-49e9-9310-a251605d06ce-oauth-serving-cert\") pod \"console-56d7c87f44-j5b8q\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") " pod="openshift-console/console-56d7c87f44-j5b8q"
Jul 14 19:48:16.434487 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.434373    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a7acfc7-7b64-49e9-9310-a251605d06ce-service-ca\") pod \"console-56d7c87f44-j5b8q\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") " pod="openshift-console/console-56d7c87f44-j5b8q"
Jul 14 19:48:16.534792 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.534763    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a7acfc7-7b64-49e9-9310-a251605d06ce-console-oauth-config\") pod \"console-56d7c87f44-j5b8q\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") " pod="openshift-console/console-56d7c87f44-j5b8q"
Jul 14 19:48:16.534948 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.534798    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a7acfc7-7b64-49e9-9310-a251605d06ce-trusted-ca-bundle\") pod \"console-56d7c87f44-j5b8q\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") " pod="openshift-console/console-56d7c87f44-j5b8q"
Jul 14 19:48:16.534948 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.534826    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a7acfc7-7b64-49e9-9310-a251605d06ce-console-config\") pod \"console-56d7c87f44-j5b8q\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") " pod="openshift-console/console-56d7c87f44-j5b8q"
Jul 14 19:48:16.535032 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.534951    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a7acfc7-7b64-49e9-9310-a251605d06ce-oauth-serving-cert\") pod \"console-56d7c87f44-j5b8q\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") " pod="openshift-console/console-56d7c87f44-j5b8q"
Jul 14 19:48:16.535032 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.534984    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a7acfc7-7b64-49e9-9310-a251605d06ce-service-ca\") pod \"console-56d7c87f44-j5b8q\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") " pod="openshift-console/console-56d7c87f44-j5b8q"
Jul 14 19:48:16.535105 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.535031    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp5ks\" (UniqueName: \"kubernetes.io/projected/4a7acfc7-7b64-49e9-9310-a251605d06ce-kube-api-access-dp5ks\") pod \"console-56d7c87f44-j5b8q\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") " pod="openshift-console/console-56d7c87f44-j5b8q"
Jul 14 19:48:16.535152 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.535110    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a7acfc7-7b64-49e9-9310-a251605d06ce-console-serving-cert\") pod \"console-56d7c87f44-j5b8q\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") " pod="openshift-console/console-56d7c87f44-j5b8q"
Jul 14 19:48:16.535667 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.535636    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a7acfc7-7b64-49e9-9310-a251605d06ce-console-config\") pod \"console-56d7c87f44-j5b8q\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") " pod="openshift-console/console-56d7c87f44-j5b8q"
Jul 14 19:48:16.535804 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.535639    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a7acfc7-7b64-49e9-9310-a251605d06ce-service-ca\") pod \"console-56d7c87f44-j5b8q\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") " pod="openshift-console/console-56d7c87f44-j5b8q"
Jul 14 19:48:16.535804 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.535693    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a7acfc7-7b64-49e9-9310-a251605d06ce-oauth-serving-cert\") pod \"console-56d7c87f44-j5b8q\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") " pod="openshift-console/console-56d7c87f44-j5b8q"
Jul 14 19:48:16.535804 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.535752    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a7acfc7-7b64-49e9-9310-a251605d06ce-trusted-ca-bundle\") pod \"console-56d7c87f44-j5b8q\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") " pod="openshift-console/console-56d7c87f44-j5b8q"
Jul 14 19:48:16.537929 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.537908    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a7acfc7-7b64-49e9-9310-a251605d06ce-console-oauth-config\") pod \"console-56d7c87f44-j5b8q\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") " pod="openshift-console/console-56d7c87f44-j5b8q"
Jul 14 19:48:16.538122 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.538103    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a7acfc7-7b64-49e9-9310-a251605d06ce-console-serving-cert\") pod \"console-56d7c87f44-j5b8q\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") " pod="openshift-console/console-56d7c87f44-j5b8q"
Jul 14 19:48:16.544433 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.544407    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp5ks\" (UniqueName: \"kubernetes.io/projected/4a7acfc7-7b64-49e9-9310-a251605d06ce-kube-api-access-dp5ks\") pod \"console-56d7c87f44-j5b8q\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") " pod="openshift-console/console-56d7c87f44-j5b8q"
Jul 14 19:48:16.730238 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:16.730198    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56d7c87f44-j5b8q"
Jul 14 19:48:18.795324 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:18.795296    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56d7c87f44-j5b8q"]
Jul 14 19:48:18.799649 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:48:18.799624    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a7acfc7_7b64_49e9_9310_a251605d06ce.slice/crio-1d643e58f947ec6aae38d368553147c7dc9c8a6405b57f24831cddf000defdfa WatchSource:0}: Error finding container 1d643e58f947ec6aae38d368553147c7dc9c8a6405b57f24831cddf000defdfa: Status 404 returned error can't find the container with id 1d643e58f947ec6aae38d368553147c7dc9c8a6405b57f24831cddf000defdfa
Jul 14 19:48:19.088399 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:19.088367    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56d7c87f44-j5b8q" event={"ID":"4a7acfc7-7b64-49e9-9310-a251605d06ce","Type":"ContainerStarted","Data":"1d643e58f947ec6aae38d368553147c7dc9c8a6405b57f24831cddf000defdfa"}
Jul 14 19:48:19.089988 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:19.089962    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fd6mw" event={"ID":"86c2f79a-dd6d-46b1-ac5c-e1d00a7a652b","Type":"ContainerStarted","Data":"d69971a5d6be014b9ba9ba60c82d2668ba14b141420cbb184d6b272578f17730"}
Jul 14 19:48:19.091827 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:19.091804    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mgqdj" event={"ID":"8fafc082-f569-4875-8d12-db7504865e6d","Type":"ContainerStarted","Data":"c0f12b2f983810e0a5114868673c66b11ef652800c0c4da4a7e6a9289425554e"}
Jul 14 19:48:19.091935 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:19.091855    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mgqdj"
Jul 14 19:48:19.091935 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:19.091883    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mgqdj" event={"ID":"8fafc082-f569-4875-8d12-db7504865e6d","Type":"ContainerStarted","Data":"6ae6518b09b5b038b59be1852659b480d085327f3d1dcfd1c2ce762ff711dfdf"}
Jul 14 19:48:19.106735 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:19.106562    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fd6mw" podStartSLOduration=1.333254076 podStartE2EDuration="7.106545224s" totalImagesPullingTime="5.773291148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:48:19.105442198 +0000 UTC m=+51.705745902" watchObservedRunningTime="2026-07-14 19:48:19.106545224 +0000 UTC m=+51.706848904"
Jul 14 19:48:19.122737 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:19.122677    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mgqdj" podStartSLOduration=1.350774134 podStartE2EDuration="7.122655132s" totalImagesPullingTime="5.771880998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:48:19.121676426 +0000 UTC m=+51.721980128" watchObservedRunningTime="2026-07-14 19:48:19.122655132 +0000 UTC m=+51.722958836"
Jul 14 19:48:20.098217 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:20.098167    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d2d6k" event={"ID":"1fd75dce-40e5-494c-92c4-eac34f1fc302","Type":"ContainerStarted","Data":"5aef638af87e65de36c18dfcd434ada88f3d865551f60e32cb09789409d9a7d4"}
Jul 14 19:48:22.106032 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:22.105989    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56d7c87f44-j5b8q" event={"ID":"4a7acfc7-7b64-49e9-9310-a251605d06ce","Type":"ContainerStarted","Data":"6c5a667614d9bbf7939f0133662c6743c3398e6658710ec0748a166952cb2601"}
Jul 14 19:48:22.124421 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:22.124322    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56d7c87f44-j5b8q" podStartSLOduration=3.032708128 podStartE2EDuration="6.124288303s" totalImagesPullingTime="3.091580175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:48:22.123918076 +0000 UTC m=+54.724221762" watchObservedRunningTime="2026-07-14 19:48:22.124288303 +0000 UTC m=+54.724592005"
Jul 14 19:48:23.111676 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:23.111635    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d2d6k" event={"ID":"1fd75dce-40e5-494c-92c4-eac34f1fc302","Type":"ContainerStarted","Data":"c954b3c99f8d93efa7c5dc9107bbb7cf832158bf003ce56753582b5046b0a2c4"}
Jul 14 19:48:25.120681 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:25.120643    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d2d6k" event={"ID":"1fd75dce-40e5-494c-92c4-eac34f1fc302","Type":"ContainerStarted","Data":"816fd23319d9807a0010b949f12a34137e318dcb5ff04f65b2d93134865eca36"}
Jul 14 19:48:25.139483 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:25.139430    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-d2d6k" podStartSLOduration=1.5882866839999998 podStartE2EDuration="13.139413364s" totalImagesPullingTime="11.55112668s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:48:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:48:25.138058861 +0000 UTC m=+57.738362566" watchObservedRunningTime="2026-07-14 19:48:25.139413364 +0000 UTC m=+57.739717086"
Jul 14 19:48:26.731331 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:26.731283    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56d7c87f44-j5b8q"
Jul 14 19:48:26.731724 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:26.731356    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56d7c87f44-j5b8q"
Jul 14 19:48:26.732619 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:26.732599    2560 patch_prober.go:28] interesting pod/console-56d7c87f44-j5b8q container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.9:8443/health\": dial tcp 10.133.0.9:8443: connect: connection refused" start-of-body=
Jul 14 19:48:26.732679 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:26.732641    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-56d7c87f44-j5b8q" podUID="4a7acfc7-7b64-49e9-9310-a251605d06ce" containerName="console" probeResult="failure" output="Get \"https://10.133.0.9:8443/health\": dial tcp 10.133.0.9:8443: connect: connection refused"
Jul 14 19:48:27.750320 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.750289    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-624qf"]
Jul 14 19:48:27.777638 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.777610    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-624qf"]
Jul 14 19:48:27.777777 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.777723    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-624qf"
Jul 14 19:48:27.780563 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.780536    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 14 19:48:27.780708 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.780571    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 14 19:48:27.781724 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.781695    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 14 19:48:27.782017 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.782001    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 14 19:48:27.782111 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.782095    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 14 19:48:27.782211 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.782197    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xdxbz\""
Jul 14 19:48:27.788042 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.788021    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gnrjv"]
Jul 14 19:48:27.804257 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.804236    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:27.806568 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.806552    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 14 19:48:27.806966 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.806951    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 14 19:48:27.807020 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.806968    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 14 19:48:27.807056 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.806951    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4l8n4\""
Jul 14 19:48:27.827664 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.827640    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc4584b4-2397-4ec4-9cf4-8f2edfa2151f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-624qf\" (UID: \"dc4584b4-2397-4ec4-9cf4-8f2edfa2151f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-624qf"
Jul 14 19:48:27.827780 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.827670    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc4584b4-2397-4ec4-9cf4-8f2edfa2151f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-624qf\" (UID: \"dc4584b4-2397-4ec4-9cf4-8f2edfa2151f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-624qf"
Jul 14 19:48:27.827780 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.827694    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc4584b4-2397-4ec4-9cf4-8f2edfa2151f-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-624qf\" (UID: \"dc4584b4-2397-4ec4-9cf4-8f2edfa2151f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-624qf"
Jul 14 19:48:27.827865 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.827816    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vslbz\" (UniqueName: \"kubernetes.io/projected/dc4584b4-2397-4ec4-9cf4-8f2edfa2151f-kube-api-access-vslbz\") pod \"openshift-state-metrics-84bd68b896-624qf\" (UID: \"dc4584b4-2397-4ec4-9cf4-8f2edfa2151f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-624qf"
Jul 14 19:48:27.928166 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.928131    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f919f1fa-9d2a-4698-b875-63205715fa3c-node-exporter-accelerators-collector-config\") pod \"node-exporter-gnrjv\" (UID: \"f919f1fa-9d2a-4698-b875-63205715fa3c\") " pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:27.928359 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.928190    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f919f1fa-9d2a-4698-b875-63205715fa3c-sys\") pod \"node-exporter-gnrjv\" (UID: \"f919f1fa-9d2a-4698-b875-63205715fa3c\") " pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:27.928359 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.928238    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f919f1fa-9d2a-4698-b875-63205715fa3c-node-exporter-textfile\") pod \"node-exporter-gnrjv\" (UID: \"f919f1fa-9d2a-4698-b875-63205715fa3c\") " pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:27.928359 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.928285    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blrlq\" (UniqueName: \"kubernetes.io/projected/f919f1fa-9d2a-4698-b875-63205715fa3c-kube-api-access-blrlq\") pod \"node-exporter-gnrjv\" (UID: \"f919f1fa-9d2a-4698-b875-63205715fa3c\") " pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:27.928359 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.928310    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc4584b4-2397-4ec4-9cf4-8f2edfa2151f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-624qf\" (UID: \"dc4584b4-2397-4ec4-9cf4-8f2edfa2151f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-624qf"
Jul 14 19:48:27.928359 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.928335    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f919f1fa-9d2a-4698-b875-63205715fa3c-node-exporter-tls\") pod \"node-exporter-gnrjv\" (UID: \"f919f1fa-9d2a-4698-b875-63205715fa3c\") " pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:27.928552 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.928373    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f919f1fa-9d2a-4698-b875-63205715fa3c-metrics-client-ca\") pod \"node-exporter-gnrjv\" (UID: \"f919f1fa-9d2a-4698-b875-63205715fa3c\") " pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:27.928552 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:27.928432    2560 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 14 19:48:27.928552 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.928460    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f919f1fa-9d2a-4698-b875-63205715fa3c-node-exporter-wtmp\") pod \"node-exporter-gnrjv\" (UID: \"f919f1fa-9d2a-4698-b875-63205715fa3c\") " pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:27.928552 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:27.928480    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc4584b4-2397-4ec4-9cf4-8f2edfa2151f-openshift-state-metrics-tls podName:dc4584b4-2397-4ec4-9cf4-8f2edfa2151f nodeName:}" failed. No retries permitted until 2026-07-14 19:48:28.428467942 +0000 UTC m=+61.028771622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/dc4584b4-2397-4ec4-9cf4-8f2edfa2151f-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-624qf" (UID: "dc4584b4-2397-4ec4-9cf4-8f2edfa2151f") : secret "openshift-state-metrics-tls" not found
Jul 14 19:48:27.928552 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.928516    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vslbz\" (UniqueName: \"kubernetes.io/projected/dc4584b4-2397-4ec4-9cf4-8f2edfa2151f-kube-api-access-vslbz\") pod \"openshift-state-metrics-84bd68b896-624qf\" (UID: \"dc4584b4-2397-4ec4-9cf4-8f2edfa2151f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-624qf"
Jul 14 19:48:27.928552 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.928546    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc4584b4-2397-4ec4-9cf4-8f2edfa2151f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-624qf\" (UID: \"dc4584b4-2397-4ec4-9cf4-8f2edfa2151f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-624qf"
Jul 14 19:48:27.928820 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.928572    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc4584b4-2397-4ec4-9cf4-8f2edfa2151f-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-624qf\" (UID: \"dc4584b4-2397-4ec4-9cf4-8f2edfa2151f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-624qf"
Jul 14 19:48:27.928820 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.928590    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f919f1fa-9d2a-4698-b875-63205715fa3c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gnrjv\" (UID: \"f919f1fa-9d2a-4698-b875-63205715fa3c\") " pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:27.928820 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.928609    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f919f1fa-9d2a-4698-b875-63205715fa3c-root\") pod \"node-exporter-gnrjv\" (UID: \"f919f1fa-9d2a-4698-b875-63205715fa3c\") " pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:27.929246 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.929223    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc4584b4-2397-4ec4-9cf4-8f2edfa2151f-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-624qf\" (UID: \"dc4584b4-2397-4ec4-9cf4-8f2edfa2151f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-624qf"
Jul 14 19:48:27.932272 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.932248    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc4584b4-2397-4ec4-9cf4-8f2edfa2151f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-624qf\" (UID: \"dc4584b4-2397-4ec4-9cf4-8f2edfa2151f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-624qf"
Jul 14 19:48:27.941754 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:27.941734    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vslbz\" (UniqueName: \"kubernetes.io/projected/dc4584b4-2397-4ec4-9cf4-8f2edfa2151f-kube-api-access-vslbz\") pod \"openshift-state-metrics-84bd68b896-624qf\" (UID: \"dc4584b4-2397-4ec4-9cf4-8f2edfa2151f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-624qf"
Jul 14 19:48:28.029017 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.028922    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f919f1fa-9d2a-4698-b875-63205715fa3c-metrics-client-ca\") pod \"node-exporter-gnrjv\" (UID: \"f919f1fa-9d2a-4698-b875-63205715fa3c\") " pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:28.029017 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.028973    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f919f1fa-9d2a-4698-b875-63205715fa3c-node-exporter-wtmp\") pod \"node-exporter-gnrjv\" (UID: \"f919f1fa-9d2a-4698-b875-63205715fa3c\") " pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:28.029017 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.029010    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f919f1fa-9d2a-4698-b875-63205715fa3c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gnrjv\" (UID: \"f919f1fa-9d2a-4698-b875-63205715fa3c\") " pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:28.029282 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.029041    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f919f1fa-9d2a-4698-b875-63205715fa3c-root\") pod \"node-exporter-gnrjv\" (UID: \"f919f1fa-9d2a-4698-b875-63205715fa3c\") " pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:28.029282 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.029087    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f919f1fa-9d2a-4698-b875-63205715fa3c-node-exporter-accelerators-collector-config\") pod \"node-exporter-gnrjv\" (UID: \"f919f1fa-9d2a-4698-b875-63205715fa3c\") " pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:28.029282 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.029123    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f919f1fa-9d2a-4698-b875-63205715fa3c-sys\") pod \"node-exporter-gnrjv\" (UID: \"f919f1fa-9d2a-4698-b875-63205715fa3c\") " pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:28.029282 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.029144    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f919f1fa-9d2a-4698-b875-63205715fa3c-node-exporter-textfile\") pod \"node-exporter-gnrjv\" (UID: \"f919f1fa-9d2a-4698-b875-63205715fa3c\") " pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:28.029282 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.029146    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f919f1fa-9d2a-4698-b875-63205715fa3c-node-exporter-wtmp\") pod \"node-exporter-gnrjv\" (UID: \"f919f1fa-9d2a-4698-b875-63205715fa3c\") " pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:28.029282 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.029160    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f919f1fa-9d2a-4698-b875-63205715fa3c-root\") pod \"node-exporter-gnrjv\" (UID: \"f919f1fa-9d2a-4698-b875-63205715fa3c\") " pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:28.029282 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.029188    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blrlq\" (UniqueName: \"kubernetes.io/projected/f919f1fa-9d2a-4698-b875-63205715fa3c-kube-api-access-blrlq\") pod \"node-exporter-gnrjv\" (UID: \"f919f1fa-9d2a-4698-b875-63205715fa3c\") " pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:28.029596 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.029409    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f919f1fa-9d2a-4698-b875-63205715fa3c-sys\") pod \"node-exporter-gnrjv\" (UID: \"f919f1fa-9d2a-4698-b875-63205715fa3c\") " pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:28.029596 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.029514    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f919f1fa-9d2a-4698-b875-63205715fa3c-metrics-client-ca\") pod \"node-exporter-gnrjv\" (UID: \"f919f1fa-9d2a-4698-b875-63205715fa3c\") " pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:28.029694 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.029598    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f919f1fa-9d2a-4698-b875-63205715fa3c-node-exporter-tls\") pod \"node-exporter-gnrjv\" (UID: \"f919f1fa-9d2a-4698-b875-63205715fa3c\") " pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:28.029995 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.029944    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f919f1fa-9d2a-4698-b875-63205715fa3c-node-exporter-accelerators-collector-config\") pod \"node-exporter-gnrjv\" (UID: \"f919f1fa-9d2a-4698-b875-63205715fa3c\") " pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:28.030125 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.030089    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f919f1fa-9d2a-4698-b875-63205715fa3c-node-exporter-textfile\") pod \"node-exporter-gnrjv\" (UID: \"f919f1fa-9d2a-4698-b875-63205715fa3c\") " pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:28.032219 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.032196    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f919f1fa-9d2a-4698-b875-63205715fa3c-node-exporter-tls\") pod \"node-exporter-gnrjv\" (UID: \"f919f1fa-9d2a-4698-b875-63205715fa3c\") " pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:28.032339 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.032281    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f919f1fa-9d2a-4698-b875-63205715fa3c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gnrjv\" (UID: \"f919f1fa-9d2a-4698-b875-63205715fa3c\") " pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:28.038402 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.038382    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blrlq\" (UniqueName: \"kubernetes.io/projected/f919f1fa-9d2a-4698-b875-63205715fa3c-kube-api-access-blrlq\") pod \"node-exporter-gnrjv\" (UID: \"f919f1fa-9d2a-4698-b875-63205715fa3c\") " pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:28.112409 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.112371    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gnrjv"
Jul 14 19:48:28.144865 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:48:28.144834    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf919f1fa_9d2a_4698_b875_63205715fa3c.slice/crio-57ca4b3cd5bc3e9ff5cd5cb1cb4272f0969c98ecddfc1386e900614f24d75257 WatchSource:0}: Error finding container 57ca4b3cd5bc3e9ff5cd5cb1cb4272f0969c98ecddfc1386e900614f24d75257: Status 404 returned error can't find the container with id 57ca4b3cd5bc3e9ff5cd5cb1cb4272f0969c98ecddfc1386e900614f24d75257
Jul 14 19:48:28.432058 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.432020    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc4584b4-2397-4ec4-9cf4-8f2edfa2151f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-624qf\" (UID: \"dc4584b4-2397-4ec4-9cf4-8f2edfa2151f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-624qf"
Jul 14 19:48:28.434502 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.434476    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc4584b4-2397-4ec4-9cf4-8f2edfa2151f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-624qf\" (UID: \"dc4584b4-2397-4ec4-9cf4-8f2edfa2151f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-624qf"
Jul 14 19:48:28.688439 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.688358    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-624qf"
Jul 14 19:48:28.835531 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.835488    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 14 19:48:28.856038 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.856004    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-624qf"]
Jul 14 19:48:28.856202 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.856061    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 14 19:48:28.856272 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.856248    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:28.856318 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:48:28.856260    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4584b4_2397_4ec4_9cf4_8f2edfa2151f.slice/crio-a3c3636348027ae43a4204502faed7424e94dd49d69574561b907d7072282a38 WatchSource:0}: Error finding container a3c3636348027ae43a4204502faed7424e94dd49d69574561b907d7072282a38: Status 404 returned error can't find the container with id a3c3636348027ae43a4204502faed7424e94dd49d69574561b907d7072282a38
Jul 14 19:48:28.858748 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.858724    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 14 19:48:28.858994 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.858978    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 14 19:48:28.859151 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.859138    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 14 19:48:28.859587 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.859450    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 14 19:48:28.859587 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.859558    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 14 19:48:28.859706 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.859691    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 14 19:48:28.859895 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.859878    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 14 19:48:28.860764 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.860746    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 14 19:48:28.860845 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.860806    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 14 19:48:28.860967 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.860952    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kjplt\""
Jul 14 19:48:28.936283 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.936255    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwjws\" (UniqueName: \"kubernetes.io/projected/889dff78-34a1-4966-8973-70bd68aa050c-kube-api-access-vwjws\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:28.936441 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.936301    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/889dff78-34a1-4966-8973-70bd68aa050c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:28.936441 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.936335    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/889dff78-34a1-4966-8973-70bd68aa050c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:28.936441 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.936415    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/889dff78-34a1-4966-8973-70bd68aa050c-web-config\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:28.936607 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.936484    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/889dff78-34a1-4966-8973-70bd68aa050c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:28.936607 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.936511    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/889dff78-34a1-4966-8973-70bd68aa050c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:28.936607 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.936542    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/889dff78-34a1-4966-8973-70bd68aa050c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:28.936715 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.936613    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/889dff78-34a1-4966-8973-70bd68aa050c-config-volume\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:28.936715 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.936661    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/889dff78-34a1-4966-8973-70bd68aa050c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:28.936715 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.936703    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/889dff78-34a1-4966-8973-70bd68aa050c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:28.936820 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.936726    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/889dff78-34a1-4966-8973-70bd68aa050c-config-out\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:28.936820 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.936749    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/889dff78-34a1-4966-8973-70bd68aa050c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:28.936820 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:28.936777    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/889dff78-34a1-4966-8973-70bd68aa050c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.037922 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.037897    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/889dff78-34a1-4966-8973-70bd68aa050c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.038073 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.037935    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/889dff78-34a1-4966-8973-70bd68aa050c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.038073 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.037965    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/889dff78-34a1-4966-8973-70bd68aa050c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.038073 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.038001    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/889dff78-34a1-4966-8973-70bd68aa050c-config-volume\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.038073 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.038038    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/889dff78-34a1-4966-8973-70bd68aa050c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.038298 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.038081    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/889dff78-34a1-4966-8973-70bd68aa050c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.038298 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.038104    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/889dff78-34a1-4966-8973-70bd68aa050c-config-out\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.038298 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.038130    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/889dff78-34a1-4966-8973-70bd68aa050c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.038298 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.038158    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/889dff78-34a1-4966-8973-70bd68aa050c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.038298 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.038219    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwjws\" (UniqueName: \"kubernetes.io/projected/889dff78-34a1-4966-8973-70bd68aa050c-kube-api-access-vwjws\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.038298 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.038259    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/889dff78-34a1-4966-8973-70bd68aa050c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.038298 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.038296    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/889dff78-34a1-4966-8973-70bd68aa050c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.038598 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.038322    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/889dff78-34a1-4966-8973-70bd68aa050c-web-config\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.039204 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.039153    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/889dff78-34a1-4966-8973-70bd68aa050c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.039617 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:29.039593    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/889dff78-34a1-4966-8973-70bd68aa050c-alertmanager-trusted-ca-bundle podName:889dff78-34a1-4966-8973-70bd68aa050c nodeName:}" failed. No retries permitted until 2026-07-14 19:48:29.539572839 +0000 UTC m=+62.139876540 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/889dff78-34a1-4966-8973-70bd68aa050c-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "889dff78-34a1-4966-8973-70bd68aa050c") : configmap references non-existent config key: ca-bundle.crt
Jul 14 19:48:29.040258 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.040234    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/889dff78-34a1-4966-8973-70bd68aa050c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.041518 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.041017    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/889dff78-34a1-4966-8973-70bd68aa050c-config-out\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.041518 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.041065    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/889dff78-34a1-4966-8973-70bd68aa050c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.041518 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.041478    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/889dff78-34a1-4966-8973-70bd68aa050c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.042119 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.041933    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/889dff78-34a1-4966-8973-70bd68aa050c-config-volume\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.042841 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.042814    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/889dff78-34a1-4966-8973-70bd68aa050c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.043015 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.042987    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/889dff78-34a1-4966-8973-70bd68aa050c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.044122 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.044100    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/889dff78-34a1-4966-8973-70bd68aa050c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.044271 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.044242    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/889dff78-34a1-4966-8973-70bd68aa050c-web-config\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.046274 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.046252    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/889dff78-34a1-4966-8973-70bd68aa050c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.050363 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.050341    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwjws\" (UniqueName: \"kubernetes.io/projected/889dff78-34a1-4966-8973-70bd68aa050c-kube-api-access-vwjws\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.100549 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.100524    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mgqdj"
Jul 14 19:48:29.135250 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.135201    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gnrjv" event={"ID":"f919f1fa-9d2a-4698-b875-63205715fa3c","Type":"ContainerStarted","Data":"57ca4b3cd5bc3e9ff5cd5cb1cb4272f0969c98ecddfc1386e900614f24d75257"}
Jul 14 19:48:29.137402 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.137371    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-624qf" event={"ID":"dc4584b4-2397-4ec4-9cf4-8f2edfa2151f","Type":"ContainerStarted","Data":"e0b3494ce3422037a2abc0d68fc1538186502fcd0ad9b379b18a651558d8b991"}
Jul 14 19:48:29.137517 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.137432    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-624qf" event={"ID":"dc4584b4-2397-4ec4-9cf4-8f2edfa2151f","Type":"ContainerStarted","Data":"45000611d1f29e873c207b4df140b6d9fc7477d71c7aaa3cc1a84be64a763d91"}
Jul 14 19:48:29.137517 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.137461    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-624qf" event={"ID":"dc4584b4-2397-4ec4-9cf4-8f2edfa2151f","Type":"ContainerStarted","Data":"a3c3636348027ae43a4204502faed7424e94dd49d69574561b907d7072282a38"}
Jul 14 19:48:29.542634 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.542597    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/889dff78-34a1-4966-8973-70bd68aa050c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.543564 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.543537    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/889dff78-34a1-4966-8973-70bd68aa050c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"889dff78-34a1-4966-8973-70bd68aa050c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.768520 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.768431    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 14 19:48:29.897845 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:29.897822    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 14 19:48:29.899925 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:48:29.899892    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod889dff78_34a1_4966_8973_70bd68aa050c.slice/crio-79174a1e9ac0e3ac620fbc7cd613b277ffecb74c906cbbadf590451782871a24 WatchSource:0}: Error finding container 79174a1e9ac0e3ac620fbc7cd613b277ffecb74c906cbbadf590451782871a24: Status 404 returned error can't find the container with id 79174a1e9ac0e3ac620fbc7cd613b277ffecb74c906cbbadf590451782871a24
Jul 14 19:48:30.035856 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:30.035832    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8tzr6"
Jul 14 19:48:30.141892 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:30.141857    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"889dff78-34a1-4966-8973-70bd68aa050c","Type":"ContainerStarted","Data":"79174a1e9ac0e3ac620fbc7cd613b277ffecb74c906cbbadf590451782871a24"}
Jul 14 19:48:30.143379 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:30.143354    2560 generic.go:356] "Generic (PLEG): container finished" podID="f919f1fa-9d2a-4698-b875-63205715fa3c" containerID="da4c3f30515c0e8e2a1d146f3ec60809d5c57dee5a862cbc81893a792db611ba" exitCode=0
Jul 14 19:48:30.143483 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:30.143390    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gnrjv" event={"ID":"f919f1fa-9d2a-4698-b875-63205715fa3c","Type":"ContainerDied","Data":"da4c3f30515c0e8e2a1d146f3ec60809d5c57dee5a862cbc81893a792db611ba"}
Jul 14 19:48:30.850778 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:30.850742    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"]
Jul 14 19:48:30.855345 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:30.855322    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:30.858807 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:30.858233    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 14 19:48:30.858807 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:30.858233    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6bzlm\""
Jul 14 19:48:30.858807 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:30.858559    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 14 19:48:30.858807 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:30.858780    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 14 19:48:30.859080 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:30.858785    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 14 19:48:30.859080 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:30.859034    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 14 19:48:30.859390 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:30.859370    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fc0fg7mfv2p40\""
Jul 14 19:48:30.866836 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:30.866559    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"]
Jul 14 19:48:30.956254 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:30.956213    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8caccfe2-d79e-4bf5-b913-e848ba5e9275-secret-thanos-querier-tls\") pod \"thanos-querier-5d79d9b6d5-frn87\" (UID: \"8caccfe2-d79e-4bf5-b913-e848ba5e9275\") " pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:30.956254 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:30.956257    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8caccfe2-d79e-4bf5-b913-e848ba5e9275-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d79d9b6d5-frn87\" (UID: \"8caccfe2-d79e-4bf5-b913-e848ba5e9275\") " pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:30.956704 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:30.956280    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8caccfe2-d79e-4bf5-b913-e848ba5e9275-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d79d9b6d5-frn87\" (UID: \"8caccfe2-d79e-4bf5-b913-e848ba5e9275\") " pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:30.956704 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:30.956389    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhfzs\" (UniqueName: \"kubernetes.io/projected/8caccfe2-d79e-4bf5-b913-e848ba5e9275-kube-api-access-zhfzs\") pod \"thanos-querier-5d79d9b6d5-frn87\" (UID: \"8caccfe2-d79e-4bf5-b913-e848ba5e9275\") " pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:30.956704 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:30.956460    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8caccfe2-d79e-4bf5-b913-e848ba5e9275-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d79d9b6d5-frn87\" (UID: \"8caccfe2-d79e-4bf5-b913-e848ba5e9275\") " pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:30.956704 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:30.956518    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8caccfe2-d79e-4bf5-b913-e848ba5e9275-metrics-client-ca\") pod \"thanos-querier-5d79d9b6d5-frn87\" (UID: \"8caccfe2-d79e-4bf5-b913-e848ba5e9275\") " pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:30.956704 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:30.956560    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8caccfe2-d79e-4bf5-b913-e848ba5e9275-secret-grpc-tls\") pod \"thanos-querier-5d79d9b6d5-frn87\" (UID: \"8caccfe2-d79e-4bf5-b913-e848ba5e9275\") " pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:30.956704 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:30.956583    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8caccfe2-d79e-4bf5-b913-e848ba5e9275-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d79d9b6d5-frn87\" (UID: \"8caccfe2-d79e-4bf5-b913-e848ba5e9275\") " pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:31.057646 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:31.057618    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8caccfe2-d79e-4bf5-b913-e848ba5e9275-secret-thanos-querier-tls\") pod \"thanos-querier-5d79d9b6d5-frn87\" (UID: \"8caccfe2-d79e-4bf5-b913-e848ba5e9275\") " pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:31.057749 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:31.057664    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8caccfe2-d79e-4bf5-b913-e848ba5e9275-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d79d9b6d5-frn87\" (UID: \"8caccfe2-d79e-4bf5-b913-e848ba5e9275\") " pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:31.057749 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:31.057709    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8caccfe2-d79e-4bf5-b913-e848ba5e9275-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d79d9b6d5-frn87\" (UID: \"8caccfe2-d79e-4bf5-b913-e848ba5e9275\") " pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:31.057855 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:31.057747    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhfzs\" (UniqueName: \"kubernetes.io/projected/8caccfe2-d79e-4bf5-b913-e848ba5e9275-kube-api-access-zhfzs\") pod \"thanos-querier-5d79d9b6d5-frn87\" (UID: \"8caccfe2-d79e-4bf5-b913-e848ba5e9275\") " pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:31.057855 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:31.057781    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8caccfe2-d79e-4bf5-b913-e848ba5e9275-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d79d9b6d5-frn87\" (UID: \"8caccfe2-d79e-4bf5-b913-e848ba5e9275\") " pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:31.057855 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:31.057813    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8caccfe2-d79e-4bf5-b913-e848ba5e9275-metrics-client-ca\") pod \"thanos-querier-5d79d9b6d5-frn87\" (UID: \"8caccfe2-d79e-4bf5-b913-e848ba5e9275\") " pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:31.057855 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:31.057850    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8caccfe2-d79e-4bf5-b913-e848ba5e9275-secret-grpc-tls\") pod \"thanos-querier-5d79d9b6d5-frn87\" (UID: \"8caccfe2-d79e-4bf5-b913-e848ba5e9275\") " pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:31.058048 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:31.057896    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8caccfe2-d79e-4bf5-b913-e848ba5e9275-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d79d9b6d5-frn87\" (UID: \"8caccfe2-d79e-4bf5-b913-e848ba5e9275\") " pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:31.058830 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:31.058773    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8caccfe2-d79e-4bf5-b913-e848ba5e9275-metrics-client-ca\") pod \"thanos-querier-5d79d9b6d5-frn87\" (UID: \"8caccfe2-d79e-4bf5-b913-e848ba5e9275\") " pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:31.060594 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:31.060568    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8caccfe2-d79e-4bf5-b913-e848ba5e9275-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d79d9b6d5-frn87\" (UID: \"8caccfe2-d79e-4bf5-b913-e848ba5e9275\") " pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:31.060865 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:31.060827    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8caccfe2-d79e-4bf5-b913-e848ba5e9275-secret-thanos-querier-tls\") pod \"thanos-querier-5d79d9b6d5-frn87\" (UID: \"8caccfe2-d79e-4bf5-b913-e848ba5e9275\") " pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:31.060865 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:31.060850    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8caccfe2-d79e-4bf5-b913-e848ba5e9275-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d79d9b6d5-frn87\" (UID: \"8caccfe2-d79e-4bf5-b913-e848ba5e9275\") " pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:31.060999 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:31.060975    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8caccfe2-d79e-4bf5-b913-e848ba5e9275-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d79d9b6d5-frn87\" (UID: \"8caccfe2-d79e-4bf5-b913-e848ba5e9275\") " pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:31.061036 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:31.060985    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8caccfe2-d79e-4bf5-b913-e848ba5e9275-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d79d9b6d5-frn87\" (UID: \"8caccfe2-d79e-4bf5-b913-e848ba5e9275\") " pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:31.061447 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:31.061426    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8caccfe2-d79e-4bf5-b913-e848ba5e9275-secret-grpc-tls\") pod \"thanos-querier-5d79d9b6d5-frn87\" (UID: \"8caccfe2-d79e-4bf5-b913-e848ba5e9275\") " pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:31.066495 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:31.066473    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhfzs\" (UniqueName: \"kubernetes.io/projected/8caccfe2-d79e-4bf5-b913-e848ba5e9275-kube-api-access-zhfzs\") pod \"thanos-querier-5d79d9b6d5-frn87\" (UID: \"8caccfe2-d79e-4bf5-b913-e848ba5e9275\") " pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:31.148927 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:31.148830    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-624qf" event={"ID":"dc4584b4-2397-4ec4-9cf4-8f2edfa2151f","Type":"ContainerStarted","Data":"2c0cb3821ab0dcb57f5f23b7be02cd801be644cacc8c018f01c1626c5e9ad331"}
Jul 14 19:48:31.150876 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:31.150850    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gnrjv" event={"ID":"f919f1fa-9d2a-4698-b875-63205715fa3c","Type":"ContainerStarted","Data":"d11fcea80e5b071a6b097bc1928d80f5091c3f2e971d79f53c754e394e27ecd5"}
Jul 14 19:48:31.150995 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:31.150901    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gnrjv" event={"ID":"f919f1fa-9d2a-4698-b875-63205715fa3c","Type":"ContainerStarted","Data":"bf25e09c1ab1c5fa2add3dde5a90b801eb4840dc0f66f6c4bb528b4bacceb59a"}
Jul 14 19:48:31.169274 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:31.169227    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-624qf" podStartSLOduration=2.299512891 podStartE2EDuration="4.169212175s" totalImagesPullingTime="1.869699284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:48:31.168006897 +0000 UTC m=+63.768310600" watchObservedRunningTime="2026-07-14 19:48:31.169212175 +0000 UTC m=+63.769515878"
Jul 14 19:48:31.169398 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:31.169366    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:31.189280 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:31.189228    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gnrjv" podStartSLOduration=2.843631127 podStartE2EDuration="4.189209522s" totalImagesPullingTime="1.27557563s" totalInitContainerRuntime="70.002765ms" isStatefulPod=true podCreationTimestamp="2026-07-14 19:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:48:31.188742533 +0000 UTC m=+63.789046236" watchObservedRunningTime="2026-07-14 19:48:31.189209522 +0000 UTC m=+63.789513227"
Jul 14 19:48:31.299408 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:31.299381    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"]
Jul 14 19:48:31.300857 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:48:31.300833    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8caccfe2_d79e_4bf5_b913_e848ba5e9275.slice/crio-6960f8a9638c6907b1a41b87618950cdebcad61c677b7d4cc00e49614e1b5bd7 WatchSource:0}: Error finding container 6960f8a9638c6907b1a41b87618950cdebcad61c677b7d4cc00e49614e1b5bd7: Status 404 returned error can't find the container with id 6960f8a9638c6907b1a41b87618950cdebcad61c677b7d4cc00e49614e1b5bd7
Jul 14 19:48:32.158205 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:32.158090    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87" event={"ID":"8caccfe2-d79e-4bf5-b913-e848ba5e9275","Type":"ContainerStarted","Data":"6960f8a9638c6907b1a41b87618950cdebcad61c677b7d4cc00e49614e1b5bd7"}
Jul 14 19:48:32.542559 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:32.542518    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56d7c87f44-j5b8q"]
Jul 14 19:48:32.573085 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:32.573055    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54d9f44c77-zd5gf"]
Jul 14 19:48:32.576162 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:32.576140    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:48:32.589267 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:32.588125    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54d9f44c77-zd5gf"]
Jul 14 19:48:32.672110 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:32.672077    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30180731-6603-4f6f-96fa-b9b78001bb6a-console-config\") pod \"console-54d9f44c77-zd5gf\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") " pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:48:32.672110 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:32.672129    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30180731-6603-4f6f-96fa-b9b78001bb6a-console-oauth-config\") pod \"console-54d9f44c77-zd5gf\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") " pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:48:32.672349 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:32.672197    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30180731-6603-4f6f-96fa-b9b78001bb6a-console-serving-cert\") pod \"console-54d9f44c77-zd5gf\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") " pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:48:32.672349 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:32.672272    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30180731-6603-4f6f-96fa-b9b78001bb6a-service-ca\") pod \"console-54d9f44c77-zd5gf\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") " pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:48:32.672419 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:32.672385    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30180731-6603-4f6f-96fa-b9b78001bb6a-trusted-ca-bundle\") pod \"console-54d9f44c77-zd5gf\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") " pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:48:32.672451 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:32.672419    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30180731-6603-4f6f-96fa-b9b78001bb6a-oauth-serving-cert\") pod \"console-54d9f44c77-zd5gf\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") " pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:48:32.672483 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:32.672462    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzkfg\" (UniqueName: \"kubernetes.io/projected/30180731-6603-4f6f-96fa-b9b78001bb6a-kube-api-access-dzkfg\") pod \"console-54d9f44c77-zd5gf\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") " pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:48:32.773472 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:32.773439    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30180731-6603-4f6f-96fa-b9b78001bb6a-console-config\") pod \"console-54d9f44c77-zd5gf\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") " pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:48:32.773472 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:32.773478    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30180731-6603-4f6f-96fa-b9b78001bb6a-console-oauth-config\") pod \"console-54d9f44c77-zd5gf\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") " pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:48:32.773747 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:32.773503    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30180731-6603-4f6f-96fa-b9b78001bb6a-console-serving-cert\") pod \"console-54d9f44c77-zd5gf\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") " pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:48:32.773747 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:32.773580    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30180731-6603-4f6f-96fa-b9b78001bb6a-service-ca\") pod \"console-54d9f44c77-zd5gf\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") " pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:48:32.773747 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:32.773661    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30180731-6603-4f6f-96fa-b9b78001bb6a-trusted-ca-bundle\") pod \"console-54d9f44c77-zd5gf\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") " pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:48:32.773747 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:32.773687    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30180731-6603-4f6f-96fa-b9b78001bb6a-oauth-serving-cert\") pod \"console-54d9f44c77-zd5gf\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") " pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:48:32.773747 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:32.773721    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzkfg\" (UniqueName: \"kubernetes.io/projected/30180731-6603-4f6f-96fa-b9b78001bb6a-kube-api-access-dzkfg\") pod \"console-54d9f44c77-zd5gf\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") " pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:48:32.774306 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:32.774271    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30180731-6603-4f6f-96fa-b9b78001bb6a-console-config\") pod \"console-54d9f44c77-zd5gf\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") " pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:48:32.774656 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:32.774632    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30180731-6603-4f6f-96fa-b9b78001bb6a-service-ca\") pod \"console-54d9f44c77-zd5gf\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") " pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:48:32.774772 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:32.774643    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30180731-6603-4f6f-96fa-b9b78001bb6a-trusted-ca-bundle\") pod \"console-54d9f44c77-zd5gf\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") " pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:48:32.775040 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:32.775015    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30180731-6603-4f6f-96fa-b9b78001bb6a-oauth-serving-cert\") pod \"console-54d9f44c77-zd5gf\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") " pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:48:32.778201 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:32.776618    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30180731-6603-4f6f-96fa-b9b78001bb6a-console-oauth-config\") pod \"console-54d9f44c77-zd5gf\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") " pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:48:32.778201 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:32.776779    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30180731-6603-4f6f-96fa-b9b78001bb6a-console-serving-cert\") pod \"console-54d9f44c77-zd5gf\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") " pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:48:32.782434 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:32.782392    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzkfg\" (UniqueName: \"kubernetes.io/projected/30180731-6603-4f6f-96fa-b9b78001bb6a-kube-api-access-dzkfg\") pod \"console-54d9f44c77-zd5gf\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") " pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:48:32.887306 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:32.887221    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:48:33.038006 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:33.037970    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54d9f44c77-zd5gf"]
Jul 14 19:48:33.228532 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:48:33.228464    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30180731_6603_4f6f_96fa_b9b78001bb6a.slice/crio-3fd8ae7b1569383301b4cabf5999527f5e2cbacf4e5a71c310b41df80b8fcb76 WatchSource:0}: Error finding container 3fd8ae7b1569383301b4cabf5999527f5e2cbacf4e5a71c310b41df80b8fcb76: Status 404 returned error can't find the container with id 3fd8ae7b1569383301b4cabf5999527f5e2cbacf4e5a71c310b41df80b8fcb76
Jul 14 19:48:33.580942 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:33.580902    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/108c0b81-209c-46de-8d27-aba218077646-metrics-certs\") pod \"network-metrics-daemon-z2mcj\" (UID: \"108c0b81-209c-46de-8d27-aba218077646\") " pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:48:33.584147 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:33.584121    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 14 19:48:33.594609 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:33.594579    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/108c0b81-209c-46de-8d27-aba218077646-metrics-certs\") pod \"network-metrics-daemon-z2mcj\" (UID: \"108c0b81-209c-46de-8d27-aba218077646\") " pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:48:33.682076 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:33.682031    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/120711d8-0534-470b-ab0f-5433b06fef47-original-pull-secret\") pod \"global-pull-secret-syncer-pp59q\" (UID: \"120711d8-0534-470b-ab0f-5433b06fef47\") " pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:48:33.684790 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:33.684761    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 14 19:48:33.698783 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:33.698754    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/120711d8-0534-470b-ab0f-5433b06fef47-original-pull-secret\") pod \"global-pull-secret-syncer-pp59q\" (UID: \"120711d8-0534-470b-ab0f-5433b06fef47\") " pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:48:33.700828 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:33.700638    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp59q"
Jul 14 19:48:33.715264 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:33.715243    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b6fln\""
Jul 14 19:48:33.723483 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:33.723299    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z2mcj"
Jul 14 19:48:33.782993 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:33.782966    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncmfq\" (UniqueName: \"kubernetes.io/projected/a11b1aab-78a3-48da-904c-71226e9c85ff-kube-api-access-ncmfq\") pod \"network-check-target-z785n\" (UID: \"a11b1aab-78a3-48da-904c-71226e9c85ff\") " pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:48:33.785784 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:33.785731    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 14 19:48:33.796802 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:33.796777    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 14 19:48:33.807118 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:33.807067    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncmfq\" (UniqueName: \"kubernetes.io/projected/a11b1aab-78a3-48da-904c-71226e9c85ff-kube-api-access-ncmfq\") pod \"network-check-target-z785n\" (UID: \"a11b1aab-78a3-48da-904c-71226e9c85ff\") " pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:48:33.951728 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:33.951643    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z2mcj"]
Jul 14 19:48:33.968209 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:48:33.968150    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod108c0b81_209c_46de_8d27_aba218077646.slice/crio-c693da1a0037f83b731c02d07b98419b0ee3ab2d9c71c8e8ca5317089c5120b1 WatchSource:0}: Error finding container c693da1a0037f83b731c02d07b98419b0ee3ab2d9c71c8e8ca5317089c5120b1: Status 404 returned error can't find the container with id c693da1a0037f83b731c02d07b98419b0ee3ab2d9c71c8e8ca5317089c5120b1
Jul 14 19:48:33.980772 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:33.980749    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pp59q"]
Jul 14 19:48:33.984904 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:48:33.984881    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod120711d8_0534_470b_ab0f_5433b06fef47.slice/crio-751f833103d2fd360698a18efd59c95b8f0691bda87557d79e31d10a086b9e82 WatchSource:0}: Error finding container 751f833103d2fd360698a18efd59c95b8f0691bda87557d79e31d10a086b9e82: Status 404 returned error can't find the container with id 751f833103d2fd360698a18efd59c95b8f0691bda87557d79e31d10a086b9e82
Jul 14 19:48:34.022811 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.022760    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-45mdm\""
Jul 14 19:48:34.030576 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.030555    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:48:34.068053 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.068018    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 19:48:34.072421 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.072400    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.079183 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.079134    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 14 19:48:34.080083 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.079429    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 14 19:48:34.080083 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.079500    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 14 19:48:34.080083 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.079638    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 14 19:48:34.080083 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.079736    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 14 19:48:34.080083 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.079431    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 14 19:48:34.080083 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.079932    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 14 19:48:34.080083 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.079984    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 14 19:48:34.080494 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.080335    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 14 19:48:34.080568 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.080548    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 14 19:48:34.081706 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.080653    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 14 19:48:34.081706 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.080801    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4tjvq52vhosp9\""
Jul 14 19:48:34.081706 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.080858    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-m6wtt\""
Jul 14 19:48:34.081706 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.080909    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 14 19:48:34.082539 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.082516    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 14 19:48:34.090886 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.090862    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 14 19:48:34.093073 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.093047    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 14 19:48:34.101981 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.101881    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 19:48:34.166875 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.166848    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z2mcj" event={"ID":"108c0b81-209c-46de-8d27-aba218077646","Type":"ContainerStarted","Data":"c693da1a0037f83b731c02d07b98419b0ee3ab2d9c71c8e8ca5317089c5120b1"}
Jul 14 19:48:34.168160 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.168136    2560 generic.go:356] "Generic (PLEG): container finished" podID="889dff78-34a1-4966-8973-70bd68aa050c" containerID="a8095a757f6967d8a45616c341e5a080df8bbd9cbf285e5fb47e1efef28309ee" exitCode=0
Jul 14 19:48:34.168302 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.168168    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"889dff78-34a1-4966-8973-70bd68aa050c","Type":"ContainerDied","Data":"a8095a757f6967d8a45616c341e5a080df8bbd9cbf285e5fb47e1efef28309ee"}
Jul 14 19:48:34.170379 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.170351    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d9f44c77-zd5gf" event={"ID":"30180731-6603-4f6f-96fa-b9b78001bb6a","Type":"ContainerStarted","Data":"1f47e076a34759ee69f1155c4753ba471e5f9e2e90cf0493244ed6640255a02b"}
Jul 14 19:48:34.170481 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.170409    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d9f44c77-zd5gf" event={"ID":"30180731-6603-4f6f-96fa-b9b78001bb6a","Type":"ContainerStarted","Data":"3fd8ae7b1569383301b4cabf5999527f5e2cbacf4e5a71c310b41df80b8fcb76"}
Jul 14 19:48:34.171580 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.171563    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z785n"]
Jul 14 19:48:34.172957 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.172924    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87" event={"ID":"8caccfe2-d79e-4bf5-b913-e848ba5e9275","Type":"ContainerStarted","Data":"eac779dbd366e14927576aac2e9e9f75663c03767b8941d5d36ab55b79733a67"}
Jul 14 19:48:34.173026 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.172997    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87" event={"ID":"8caccfe2-d79e-4bf5-b913-e848ba5e9275","Type":"ContainerStarted","Data":"59242f13db675b78e68a4fcf365a7042c85d31e1d3abd607b4c06de6ae65ff4a"}
Jul 14 19:48:34.173124 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.173033    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87" event={"ID":"8caccfe2-d79e-4bf5-b913-e848ba5e9275","Type":"ContainerStarted","Data":"e15830ccb83c2438484167b9e60a82e43ec3c32bea9555084592cab01df7d6e5"}
Jul 14 19:48:34.174065 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.174041    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pp59q" event={"ID":"120711d8-0534-470b-ab0f-5433b06fef47","Type":"ContainerStarted","Data":"751f833103d2fd360698a18efd59c95b8f0691bda87557d79e31d10a086b9e82"}
Jul 14 19:48:34.175096 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:48:34.175075    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11b1aab_78a3_48da_904c_71226e9c85ff.slice/crio-c894638fbd319b7d889efabee2959e932fcc0e09fd602b1265d966823f003f52 WatchSource:0}: Error finding container c894638fbd319b7d889efabee2959e932fcc0e09fd602b1265d966823f003f52: Status 404 returned error can't find the container with id c894638fbd319b7d889efabee2959e932fcc0e09fd602b1265d966823f003f52
Jul 14 19:48:34.186717 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.186695    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6d8962ff-61b2-4a31-a784-dfe49041f886-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.186808 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.186733    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6d8962ff-61b2-4a31-a784-dfe49041f886-config-out\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.186808 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.186755    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d8962ff-61b2-4a31-a784-dfe49041f886-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.186808 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.186780    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6d8962ff-61b2-4a31-a784-dfe49041f886-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.186808 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.186803    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6d8962ff-61b2-4a31-a784-dfe49041f886-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.186954 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.186858    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d8962ff-61b2-4a31-a784-dfe49041f886-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.186954 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.186903    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d8962ff-61b2-4a31-a784-dfe49041f886-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.186954 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.186933    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6d8962ff-61b2-4a31-a784-dfe49041f886-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.187062 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.186963    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6d8962ff-61b2-4a31-a784-dfe49041f886-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.187062 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.186990    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zt4h\" (UniqueName: \"kubernetes.io/projected/6d8962ff-61b2-4a31-a784-dfe49041f886-kube-api-access-5zt4h\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.187062 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.187021    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6d8962ff-61b2-4a31-a784-dfe49041f886-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.187062 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.187050    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6d8962ff-61b2-4a31-a784-dfe49041f886-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.187260 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.187075    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6d8962ff-61b2-4a31-a784-dfe49041f886-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.187260 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.187109    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6d8962ff-61b2-4a31-a784-dfe49041f886-web-config\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.187260 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.187161    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6d8962ff-61b2-4a31-a784-dfe49041f886-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.187260 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.187215    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6d8962ff-61b2-4a31-a784-dfe49041f886-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.187260 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.187247    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d8962ff-61b2-4a31-a784-dfe49041f886-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.187464 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.187284    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6d8962ff-61b2-4a31-a784-dfe49041f886-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.187464 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.187309    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6d8962ff-61b2-4a31-a784-dfe49041f886-config\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.187464 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.187338    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6d8962ff-61b2-4a31-a784-dfe49041f886-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.232309 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.232262    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54d9f44c77-zd5gf" podStartSLOduration=2.232245931 podStartE2EDuration="2.232245931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:48:34.231157399 +0000 UTC m=+66.831461102" watchObservedRunningTime="2026-07-14 19:48:34.232245931 +0000 UTC m=+66.832549611"
Jul 14 19:48:34.288229 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.288160    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6d8962ff-61b2-4a31-a784-dfe49041f886-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.288414 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.288238    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6d8962ff-61b2-4a31-a784-dfe49041f886-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.288414 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.288281    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d8962ff-61b2-4a31-a784-dfe49041f886-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.288642 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.288445    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6d8962ff-61b2-4a31-a784-dfe49041f886-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.288642 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.288500    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6d8962ff-61b2-4a31-a784-dfe49041f886-config\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.288642 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.288562    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6d8962ff-61b2-4a31-a784-dfe49041f886-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.289231 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.289201    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6d8962ff-61b2-4a31-a784-dfe49041f886-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.289503 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.289480    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6d8962ff-61b2-4a31-a784-dfe49041f886-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.289721 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.289641    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d8962ff-61b2-4a31-a784-dfe49041f886-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.293803 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.292144    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6d8962ff-61b2-4a31-a784-dfe49041f886-config-out\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.293803 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.292206    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d8962ff-61b2-4a31-a784-dfe49041f886-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.293803 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.292347    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6d8962ff-61b2-4a31-a784-dfe49041f886-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.293803 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.292377    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6d8962ff-61b2-4a31-a784-dfe49041f886-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.293803 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.292432    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d8962ff-61b2-4a31-a784-dfe49041f886-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.293803 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.292484    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d8962ff-61b2-4a31-a784-dfe49041f886-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.293803 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.292624    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6d8962ff-61b2-4a31-a784-dfe49041f886-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.293803 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.292653    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6d8962ff-61b2-4a31-a784-dfe49041f886-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.293803 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.292772    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zt4h\" (UniqueName: \"kubernetes.io/projected/6d8962ff-61b2-4a31-a784-dfe49041f886-kube-api-access-5zt4h\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.293803 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.292817    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6d8962ff-61b2-4a31-a784-dfe49041f886-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.293803 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.292892    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6d8962ff-61b2-4a31-a784-dfe49041f886-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.293803 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.292992    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6d8962ff-61b2-4a31-a784-dfe49041f886-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.293803 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.293223    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6d8962ff-61b2-4a31-a784-dfe49041f886-web-config\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.295516 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.295494    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6d8962ff-61b2-4a31-a784-dfe49041f886-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.295676 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.295624    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6d8962ff-61b2-4a31-a784-dfe49041f886-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.295816 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.295785    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6d8962ff-61b2-4a31-a784-dfe49041f886-config\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.296383 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.296363    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6d8962ff-61b2-4a31-a784-dfe49041f886-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.296487 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.296449    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d8962ff-61b2-4a31-a784-dfe49041f886-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.296758 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.296732    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d8962ff-61b2-4a31-a784-dfe49041f886-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.297078 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.297053    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d8962ff-61b2-4a31-a784-dfe49041f886-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.301044 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.298494    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6d8962ff-61b2-4a31-a784-dfe49041f886-config-out\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.301044 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.298575    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6d8962ff-61b2-4a31-a784-dfe49041f886-web-config\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.301044 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.298699    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6d8962ff-61b2-4a31-a784-dfe49041f886-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.301044 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.298887    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6d8962ff-61b2-4a31-a784-dfe49041f886-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.301044 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.299840    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6d8962ff-61b2-4a31-a784-dfe49041f886-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.301044 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.300413    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6d8962ff-61b2-4a31-a784-dfe49041f886-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.301044 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.300872    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6d8962ff-61b2-4a31-a784-dfe49041f886-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.301044 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.300955    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6d8962ff-61b2-4a31-a784-dfe49041f886-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.301044 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.300975    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6d8962ff-61b2-4a31-a784-dfe49041f886-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.302033 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.302014    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6d8962ff-61b2-4a31-a784-dfe49041f886-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.303377 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.303356    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zt4h\" (UniqueName: \"kubernetes.io/projected/6d8962ff-61b2-4a31-a784-dfe49041f886-kube-api-access-5zt4h\") pod \"prometheus-k8s-0\" (UID: \"6d8962ff-61b2-4a31-a784-dfe49041f886\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.384541 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.384453    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:34.550578 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:34.550553    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 19:48:34.553070 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:48:34.553031    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8962ff_61b2_4a31_a784_dfe49041f886.slice/crio-82d9c928a7850b594e3f00b61b5f74c8e3af67309e449dbae9e2b1518cdb58b9 WatchSource:0}: Error finding container 82d9c928a7850b594e3f00b61b5f74c8e3af67309e449dbae9e2b1518cdb58b9: Status 404 returned error can't find the container with id 82d9c928a7850b594e3f00b61b5f74c8e3af67309e449dbae9e2b1518cdb58b9
Jul 14 19:48:35.180591 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:35.180520    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z785n" event={"ID":"a11b1aab-78a3-48da-904c-71226e9c85ff","Type":"ContainerStarted","Data":"c894638fbd319b7d889efabee2959e932fcc0e09fd602b1265d966823f003f52"}
Jul 14 19:48:35.182415 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:35.182387    2560 generic.go:356] "Generic (PLEG): container finished" podID="6d8962ff-61b2-4a31-a784-dfe49041f886" containerID="46ab9726668cfffdf2e38998a872c766c736d88158f426298d7c4ce221ea5238" exitCode=0
Jul 14 19:48:35.182532 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:35.182484    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6d8962ff-61b2-4a31-a784-dfe49041f886","Type":"ContainerDied","Data":"46ab9726668cfffdf2e38998a872c766c736d88158f426298d7c4ce221ea5238"}
Jul 14 19:48:35.182582 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:35.182560    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6d8962ff-61b2-4a31-a784-dfe49041f886","Type":"ContainerStarted","Data":"82d9c928a7850b594e3f00b61b5f74c8e3af67309e449dbae9e2b1518cdb58b9"}
Jul 14 19:48:36.191585 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:36.191497    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z2mcj" event={"ID":"108c0b81-209c-46de-8d27-aba218077646","Type":"ContainerStarted","Data":"bf55a03624508f0201eabf00f326bbfc54221fc61428a71ed4c6e05f43e487ff"}
Jul 14 19:48:36.191585 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:36.191566    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z2mcj" event={"ID":"108c0b81-209c-46de-8d27-aba218077646","Type":"ContainerStarted","Data":"aed392b2598c212e1e8f5f7fcec8091afe09b154cbec35e7e9ee886bfba8a0e8"}
Jul 14 19:48:36.198275 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:36.197315    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87" event={"ID":"8caccfe2-d79e-4bf5-b913-e848ba5e9275","Type":"ContainerStarted","Data":"a58962eb31978bc1d34c54c380d722022c24316d66f8fd0404ab18184f82a2aa"}
Jul 14 19:48:36.198275 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:36.197377    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87" event={"ID":"8caccfe2-d79e-4bf5-b913-e848ba5e9275","Type":"ContainerStarted","Data":"921bfd2a80688387fd5a60796b37d4683d39f42d02f9562588fb6f31b46ff2fe"}
Jul 14 19:48:36.198275 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:36.197410    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87" event={"ID":"8caccfe2-d79e-4bf5-b913-e848ba5e9275","Type":"ContainerStarted","Data":"ea39926a3d94efb6c4cd039ecc7d8e25004edaa94f9d69ab5a6086a81d03ba25"}
Jul 14 19:48:36.198275 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:36.198188    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:36.208063 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:36.208015    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-z2mcj" podStartSLOduration=66.597286626 podStartE2EDuration="1m8.207997028s" totalImagesPullingTime="1.610710402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:48:36.206834356 +0000 UTC m=+68.807138058" watchObservedRunningTime="2026-07-14 19:48:36.207997028 +0000 UTC m=+68.808300732"
Jul 14 19:48:36.230048 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:36.229552    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87" podStartSLOduration=2.206000603 podStartE2EDuration="6.229531738s" totalImagesPullingTime="4.023531135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:48:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:48:36.22711861 +0000 UTC m=+68.827422314" watchObservedRunningTime="2026-07-14 19:48:36.229531738 +0000 UTC m=+68.829835442"
Jul 14 19:48:38.218309 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:38.218219    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5d79d9b6d5-frn87"
Jul 14 19:48:41.229465 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:41.229372    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"889dff78-34a1-4966-8973-70bd68aa050c","Type":"ContainerStarted","Data":"913e48821c0f2410fc1c86229915e5edb69a5a846c7d4af78d40adf03e7d6a44"}
Jul 14 19:48:41.229925 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:41.229516    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"889dff78-34a1-4966-8973-70bd68aa050c","Type":"ContainerStarted","Data":"f4d6722ba2790f0e40db184c7f33e75ca149bc082aa38453592a4f46cf7dbcfa"}
Jul 14 19:48:41.229925 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:41.229567    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"889dff78-34a1-4966-8973-70bd68aa050c","Type":"ContainerStarted","Data":"f08cf8a332de2a76be10090ef0f9a1353a31902818a5ee38742ea75868a68dfe"}
Jul 14 19:48:41.229925 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:41.229615    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"889dff78-34a1-4966-8973-70bd68aa050c","Type":"ContainerStarted","Data":"7ba16a28c5a87f41b021bce102f8904d1abfbe1a2c9ea8fad9ee3fee7bb28f9b"}
Jul 14 19:48:41.229925 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:41.229653    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"889dff78-34a1-4966-8973-70bd68aa050c","Type":"ContainerStarted","Data":"ec92ff4001cde40e0abf97dfe223b63e14e348db88b12872b47387e5adec4523"}
Jul 14 19:48:41.229925 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:41.229689    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"889dff78-34a1-4966-8973-70bd68aa050c","Type":"ContainerStarted","Data":"2fff7991d7155efccaa3896f3a8ad60b7060ffb58c71c23d9a7b629505e6a6b5"}
Jul 14 19:48:41.230991 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:41.230968    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z785n" event={"ID":"a11b1aab-78a3-48da-904c-71226e9c85ff","Type":"ContainerStarted","Data":"856de7c5b47a066a2c0512ba627b269d9f8d81f549011235166d0f04c63e83e3"}
Jul 14 19:48:41.231095 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:41.231040    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:48:41.232443 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:41.232424    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pp59q" event={"ID":"120711d8-0534-470b-ab0f-5433b06fef47","Type":"ContainerStarted","Data":"34660ddfafa8bc815e65db5a9eae903d7faead481fff6a4ef6769c4f8f571af0"}
Jul 14 19:48:41.235631 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:41.235612    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6d8962ff-61b2-4a31-a784-dfe49041f886","Type":"ContainerStarted","Data":"9305faa0f0fccbe5a2b3525156afaec8814d058a960f69842a12060f0106327f"}
Jul 14 19:48:41.235717 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:41.235661    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6d8962ff-61b2-4a31-a784-dfe49041f886","Type":"ContainerStarted","Data":"b01a6e4655fdf3fce79d33af8d3b4f39a59f663372143f4c302b915dc7476ca1"}
Jul 14 19:48:41.235717 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:41.235685    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6d8962ff-61b2-4a31-a784-dfe49041f886","Type":"ContainerStarted","Data":"e3bafaa9c5762363fe0be22fb8f0af4ea00fc9a8570c8f542d08a26c40522138"}
Jul 14 19:48:41.235717 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:41.235707    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6d8962ff-61b2-4a31-a784-dfe49041f886","Type":"ContainerStarted","Data":"5e54c20b4fd05fc085535e1f59f363f2d746be20fb22e1f8658b4a8456cec22a"}
Jul 14 19:48:41.235804 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:41.235728    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6d8962ff-61b2-4a31-a784-dfe49041f886","Type":"ContainerStarted","Data":"423994f427d781b2de3c57ff8a318cf51a8f01626937a4bf93709230eb6a7925"}
Jul 14 19:48:41.235804 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:41.235754    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6d8962ff-61b2-4a31-a784-dfe49041f886","Type":"ContainerStarted","Data":"25dc699734e656967bed7e2d55da05447c1f8cd8a42582379bb78603ce77f3d1"}
Jul 14 19:48:41.259998 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:41.259947    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.862573057 podStartE2EDuration="13.259928447s" totalImagesPullingTime="10.368331711s" totalInitContainerRuntime="29.023679ms" isStatefulPod=false podCreationTimestamp="2026-07-14 19:48:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:48:41.256601305 +0000 UTC m=+73.856905008" watchObservedRunningTime="2026-07-14 19:48:41.259928447 +0000 UTC m=+73.860232150"
Jul 14 19:48:41.288153 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:41.288106    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.796345918 podStartE2EDuration="7.288089749s" totalImagesPullingTime="5.464098083s" totalInitContainerRuntime="27.645748ms" isStatefulPod=false podCreationTimestamp="2026-07-14 19:48:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:48:41.286714664 +0000 UTC m=+73.887018402" watchObservedRunningTime="2026-07-14 19:48:41.288089749 +0000 UTC m=+73.888393487"
Jul 14 19:48:41.302159 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:41.302116    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pp59q" podStartSLOduration=66.593682393 podStartE2EDuration="1m13.302101973s" totalImagesPullingTime="6.70841958s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:48:41.301452065 +0000 UTC m=+73.901755767" watchObservedRunningTime="2026-07-14 19:48:41.302101973 +0000 UTC m=+73.902405721"
Jul 14 19:48:41.317157 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:41.317110    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-z785n" podStartSLOduration=66.854212452 podStartE2EDuration="1m13.317092216s" totalImagesPullingTime="6.462879764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:48:41.315843655 +0000 UTC m=+73.916147359" watchObservedRunningTime="2026-07-14 19:48:41.317092216 +0000 UTC m=+73.917395919"
Jul 14 19:48:42.887929 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:42.887889    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:48:42.888334 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:42.887986    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:48:42.892698 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:42.892677    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:48:43.246525 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:43.246443    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:48:44.385196 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:44.385147    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:48:57.569828 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:57.569694    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56d7c87f44-j5b8q" podUID="4a7acfc7-7b64-49e9-9310-a251605d06ce" containerName="console" containerID="cri-o://6c5a667614d9bbf7939f0133662c6743c3398e6658710ec0748a166952cb2601" gracePeriod=15
Jul 14 19:48:57.821017 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:57.820960    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56d7c87f44-j5b8q"
Jul 14 19:48:57.903795 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:57.903764    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a7acfc7-7b64-49e9-9310-a251605d06ce-console-config\") pod \"4a7acfc7-7b64-49e9-9310-a251605d06ce\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") "
Jul 14 19:48:57.903795 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:57.903796    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a7acfc7-7b64-49e9-9310-a251605d06ce-oauth-serving-cert\") pod \"4a7acfc7-7b64-49e9-9310-a251605d06ce\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") "
Jul 14 19:48:57.904047 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:57.903833    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a7acfc7-7b64-49e9-9310-a251605d06ce-service-ca\") pod \"4a7acfc7-7b64-49e9-9310-a251605d06ce\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") "
Jul 14 19:48:57.904047 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:57.903888    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a7acfc7-7b64-49e9-9310-a251605d06ce-trusted-ca-bundle\") pod \"4a7acfc7-7b64-49e9-9310-a251605d06ce\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") "
Jul 14 19:48:57.904047 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:57.903931    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dp5ks\" (UniqueName: \"kubernetes.io/projected/4a7acfc7-7b64-49e9-9310-a251605d06ce-kube-api-access-dp5ks\") pod \"4a7acfc7-7b64-49e9-9310-a251605d06ce\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") "
Jul 14 19:48:57.904047 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:57.903976    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a7acfc7-7b64-49e9-9310-a251605d06ce-console-serving-cert\") pod \"4a7acfc7-7b64-49e9-9310-a251605d06ce\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") "
Jul 14 19:48:57.904047 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:57.904001    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a7acfc7-7b64-49e9-9310-a251605d06ce-console-oauth-config\") pod \"4a7acfc7-7b64-49e9-9310-a251605d06ce\" (UID: \"4a7acfc7-7b64-49e9-9310-a251605d06ce\") "
Jul 14 19:48:57.904322 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:57.904254    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a7acfc7-7b64-49e9-9310-a251605d06ce-service-ca" (OuterVolumeSpecName: "service-ca") pod "4a7acfc7-7b64-49e9-9310-a251605d06ce" (UID: "4a7acfc7-7b64-49e9-9310-a251605d06ce"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:48:57.904373 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:57.904311    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a7acfc7-7b64-49e9-9310-a251605d06ce-console-config" (OuterVolumeSpecName: "console-config") pod "4a7acfc7-7b64-49e9-9310-a251605d06ce" (UID: "4a7acfc7-7b64-49e9-9310-a251605d06ce"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:48:57.904373 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:57.904320    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a7acfc7-7b64-49e9-9310-a251605d06ce-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4a7acfc7-7b64-49e9-9310-a251605d06ce" (UID: "4a7acfc7-7b64-49e9-9310-a251605d06ce"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:48:57.904443 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:57.904413    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a7acfc7-7b64-49e9-9310-a251605d06ce-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4a7acfc7-7b64-49e9-9310-a251605d06ce" (UID: "4a7acfc7-7b64-49e9-9310-a251605d06ce"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:48:57.906686 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:57.906640    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a7acfc7-7b64-49e9-9310-a251605d06ce-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4a7acfc7-7b64-49e9-9310-a251605d06ce" (UID: "4a7acfc7-7b64-49e9-9310-a251605d06ce"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:48:57.906803 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:57.906681    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a7acfc7-7b64-49e9-9310-a251605d06ce-kube-api-access-dp5ks" (OuterVolumeSpecName: "kube-api-access-dp5ks") pod "4a7acfc7-7b64-49e9-9310-a251605d06ce" (UID: "4a7acfc7-7b64-49e9-9310-a251605d06ce"). InnerVolumeSpecName "kube-api-access-dp5ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:48:57.906803 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:57.906687    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a7acfc7-7b64-49e9-9310-a251605d06ce-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4a7acfc7-7b64-49e9-9310-a251605d06ce" (UID: "4a7acfc7-7b64-49e9-9310-a251605d06ce"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:48:58.005103 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:58.005066    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a7acfc7-7b64-49e9-9310-a251605d06ce-service-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:48:58.005103 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:58.005094    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a7acfc7-7b64-49e9-9310-a251605d06ce-trusted-ca-bundle\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:48:58.005103 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:58.005107    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp5ks\" (UniqueName: \"kubernetes.io/projected/4a7acfc7-7b64-49e9-9310-a251605d06ce-kube-api-access-dp5ks\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:48:58.005347 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:58.005118    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a7acfc7-7b64-49e9-9310-a251605d06ce-console-serving-cert\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:48:58.005347 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:58.005130    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a7acfc7-7b64-49e9-9310-a251605d06ce-console-oauth-config\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:48:58.005347 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:58.005141    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a7acfc7-7b64-49e9-9310-a251605d06ce-console-config\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:48:58.005347 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:58.005152    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a7acfc7-7b64-49e9-9310-a251605d06ce-oauth-serving-cert\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:48:58.306930 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:58.306896    2560 generic.go:356] "Generic (PLEG): container finished" podID="4a7acfc7-7b64-49e9-9310-a251605d06ce" containerID="6c5a667614d9bbf7939f0133662c6743c3398e6658710ec0748a166952cb2601" exitCode=0
Jul 14 19:48:58.307107 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:58.306947    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56d7c87f44-j5b8q"
Jul 14 19:48:58.307107 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:58.306955    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56d7c87f44-j5b8q" event={"ID":"4a7acfc7-7b64-49e9-9310-a251605d06ce","Type":"ContainerDied","Data":"6c5a667614d9bbf7939f0133662c6743c3398e6658710ec0748a166952cb2601"}
Jul 14 19:48:58.307107 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:58.306999    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56d7c87f44-j5b8q" event={"ID":"4a7acfc7-7b64-49e9-9310-a251605d06ce","Type":"ContainerDied","Data":"1d643e58f947ec6aae38d368553147c7dc9c8a6405b57f24831cddf000defdfa"}
Jul 14 19:48:58.307107 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:58.307018    2560 scope.go:117] "RemoveContainer" containerID="6c5a667614d9bbf7939f0133662c6743c3398e6658710ec0748a166952cb2601"
Jul 14 19:48:58.315979 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:58.315960    2560 scope.go:117] "RemoveContainer" containerID="6c5a667614d9bbf7939f0133662c6743c3398e6658710ec0748a166952cb2601"
Jul 14 19:48:58.316282 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:48:58.316256    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c5a667614d9bbf7939f0133662c6743c3398e6658710ec0748a166952cb2601\": container with ID starting with 6c5a667614d9bbf7939f0133662c6743c3398e6658710ec0748a166952cb2601 not found: ID does not exist" containerID="6c5a667614d9bbf7939f0133662c6743c3398e6658710ec0748a166952cb2601"
Jul 14 19:48:58.316334 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:58.316290    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5a667614d9bbf7939f0133662c6743c3398e6658710ec0748a166952cb2601"} err="failed to get container status \"6c5a667614d9bbf7939f0133662c6743c3398e6658710ec0748a166952cb2601\": rpc error: code = NotFound desc = could not find container \"6c5a667614d9bbf7939f0133662c6743c3398e6658710ec0748a166952cb2601\": container with ID starting with 6c5a667614d9bbf7939f0133662c6743c3398e6658710ec0748a166952cb2601 not found: ID does not exist"
Jul 14 19:48:58.328994 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:58.328968    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56d7c87f44-j5b8q"]
Jul 14 19:48:58.333117 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:58.333097    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56d7c87f44-j5b8q"]
Jul 14 19:48:59.888925 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:48:59.888886    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a7acfc7-7b64-49e9-9310-a251605d06ce" path="/var/lib/kubelet/pods/4a7acfc7-7b64-49e9-9310-a251605d06ce/volumes"
Jul 14 19:49:12.242324 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:12.242296    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-z785n"
Jul 14 19:49:34.384762 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:34.384718    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:49:34.405150 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:34.405119    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:49:34.466717 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:34.466687    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 19:49:57.968361 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:57.968326    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56db7c8bf4-9dr8h"]
Jul 14 19:49:57.968822 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:57.968616    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a7acfc7-7b64-49e9-9310-a251605d06ce" containerName="console"
Jul 14 19:49:57.968822 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:57.968628    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a7acfc7-7b64-49e9-9310-a251605d06ce" containerName="console"
Jul 14 19:49:57.968822 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:57.968686    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a7acfc7-7b64-49e9-9310-a251605d06ce" containerName="console"
Jul 14 19:49:57.970360 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:57.970344    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56db7c8bf4-9dr8h"
Jul 14 19:49:57.987529 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:57.987507    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56db7c8bf4-9dr8h"]
Jul 14 19:49:58.005513 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:58.005492    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59a98800-4746-4bd4-bfcf-afc686b2dafa-console-config\") pod \"console-56db7c8bf4-9dr8h\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") " pod="openshift-console/console-56db7c8bf4-9dr8h"
Jul 14 19:49:58.005629 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:58.005523    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59a98800-4746-4bd4-bfcf-afc686b2dafa-service-ca\") pod \"console-56db7c8bf4-9dr8h\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") " pod="openshift-console/console-56db7c8bf4-9dr8h"
Jul 14 19:49:58.005629 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:58.005540    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59a98800-4746-4bd4-bfcf-afc686b2dafa-trusted-ca-bundle\") pod \"console-56db7c8bf4-9dr8h\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") " pod="openshift-console/console-56db7c8bf4-9dr8h"
Jul 14 19:49:58.005629 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:58.005567    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkdbn\" (UniqueName: \"kubernetes.io/projected/59a98800-4746-4bd4-bfcf-afc686b2dafa-kube-api-access-qkdbn\") pod \"console-56db7c8bf4-9dr8h\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") " pod="openshift-console/console-56db7c8bf4-9dr8h"
Jul 14 19:49:58.005629 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:58.005615    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59a98800-4746-4bd4-bfcf-afc686b2dafa-oauth-serving-cert\") pod \"console-56db7c8bf4-9dr8h\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") " pod="openshift-console/console-56db7c8bf4-9dr8h"
Jul 14 19:49:58.005766 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:58.005635    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59a98800-4746-4bd4-bfcf-afc686b2dafa-console-oauth-config\") pod \"console-56db7c8bf4-9dr8h\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") " pod="openshift-console/console-56db7c8bf4-9dr8h"
Jul 14 19:49:58.005766 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:58.005688    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59a98800-4746-4bd4-bfcf-afc686b2dafa-console-serving-cert\") pod \"console-56db7c8bf4-9dr8h\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") " pod="openshift-console/console-56db7c8bf4-9dr8h"
Jul 14 19:49:58.106193 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:58.106140    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59a98800-4746-4bd4-bfcf-afc686b2dafa-oauth-serving-cert\") pod \"console-56db7c8bf4-9dr8h\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") " pod="openshift-console/console-56db7c8bf4-9dr8h"
Jul 14 19:49:58.106392 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:58.106207    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59a98800-4746-4bd4-bfcf-afc686b2dafa-console-oauth-config\") pod \"console-56db7c8bf4-9dr8h\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") " pod="openshift-console/console-56db7c8bf4-9dr8h"
Jul 14 19:49:58.106392 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:58.106234    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59a98800-4746-4bd4-bfcf-afc686b2dafa-console-serving-cert\") pod \"console-56db7c8bf4-9dr8h\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") " pod="openshift-console/console-56db7c8bf4-9dr8h"
Jul 14 19:49:58.106392 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:58.106269    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59a98800-4746-4bd4-bfcf-afc686b2dafa-console-config\") pod \"console-56db7c8bf4-9dr8h\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") " pod="openshift-console/console-56db7c8bf4-9dr8h"
Jul 14 19:49:58.106392 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:58.106304    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59a98800-4746-4bd4-bfcf-afc686b2dafa-service-ca\") pod \"console-56db7c8bf4-9dr8h\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") " pod="openshift-console/console-56db7c8bf4-9dr8h"
Jul 14 19:49:58.106392 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:58.106327    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59a98800-4746-4bd4-bfcf-afc686b2dafa-trusted-ca-bundle\") pod \"console-56db7c8bf4-9dr8h\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") " pod="openshift-console/console-56db7c8bf4-9dr8h"
Jul 14 19:49:58.106392 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:58.106366    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkdbn\" (UniqueName: \"kubernetes.io/projected/59a98800-4746-4bd4-bfcf-afc686b2dafa-kube-api-access-qkdbn\") pod \"console-56db7c8bf4-9dr8h\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") " pod="openshift-console/console-56db7c8bf4-9dr8h"
Jul 14 19:49:58.106956 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:58.106930    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59a98800-4746-4bd4-bfcf-afc686b2dafa-oauth-serving-cert\") pod \"console-56db7c8bf4-9dr8h\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") " pod="openshift-console/console-56db7c8bf4-9dr8h"
Jul 14 19:49:58.107063 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:58.107009    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59a98800-4746-4bd4-bfcf-afc686b2dafa-service-ca\") pod \"console-56db7c8bf4-9dr8h\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") " pod="openshift-console/console-56db7c8bf4-9dr8h"
Jul 14 19:49:58.107063 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:58.107009    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59a98800-4746-4bd4-bfcf-afc686b2dafa-console-config\") pod \"console-56db7c8bf4-9dr8h\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") " pod="openshift-console/console-56db7c8bf4-9dr8h"
Jul 14 19:49:58.107487 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:58.107460    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59a98800-4746-4bd4-bfcf-afc686b2dafa-trusted-ca-bundle\") pod \"console-56db7c8bf4-9dr8h\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") " pod="openshift-console/console-56db7c8bf4-9dr8h"
Jul 14 19:49:58.108787 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:58.108766    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59a98800-4746-4bd4-bfcf-afc686b2dafa-console-serving-cert\") pod \"console-56db7c8bf4-9dr8h\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") " pod="openshift-console/console-56db7c8bf4-9dr8h"
Jul 14 19:49:58.108885 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:58.108867    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59a98800-4746-4bd4-bfcf-afc686b2dafa-console-oauth-config\") pod \"console-56db7c8bf4-9dr8h\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") " pod="openshift-console/console-56db7c8bf4-9dr8h"
Jul 14 19:49:58.115277 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:58.115260    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkdbn\" (UniqueName: \"kubernetes.io/projected/59a98800-4746-4bd4-bfcf-afc686b2dafa-kube-api-access-qkdbn\") pod \"console-56db7c8bf4-9dr8h\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") " pod="openshift-console/console-56db7c8bf4-9dr8h"
Jul 14 19:49:58.279580 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:58.279490    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56db7c8bf4-9dr8h"
Jul 14 19:49:58.416875 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:58.416841    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56db7c8bf4-9dr8h"]
Jul 14 19:49:58.419829 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:49:58.419806    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a98800_4746_4bd4_bfcf_afc686b2dafa.slice/crio-7c947459fd2549cb3044b1c538e9e856f3f064e07e77ab4e9c02e3252e3ae25d WatchSource:0}: Error finding container 7c947459fd2549cb3044b1c538e9e856f3f064e07e77ab4e9c02e3252e3ae25d: Status 404 returned error can't find the container with id 7c947459fd2549cb3044b1c538e9e856f3f064e07e77ab4e9c02e3252e3ae25d
Jul 14 19:49:58.535702 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:58.535674    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56db7c8bf4-9dr8h" event={"ID":"59a98800-4746-4bd4-bfcf-afc686b2dafa","Type":"ContainerStarted","Data":"191fbc93ab17a2f0f4c1d0327f405326eb031d68a3d02fe8fdc80552af67a283"}
Jul 14 19:49:58.535861 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:58.535726    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56db7c8bf4-9dr8h" event={"ID":"59a98800-4746-4bd4-bfcf-afc686b2dafa","Type":"ContainerStarted","Data":"7c947459fd2549cb3044b1c538e9e856f3f064e07e77ab4e9c02e3252e3ae25d"}
Jul 14 19:49:58.553059 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:49:58.553005    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56db7c8bf4-9dr8h" podStartSLOduration=1.552984737 podStartE2EDuration="1.552984737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:49:58.55294223 +0000 UTC m=+151.153245934" watchObservedRunningTime="2026-07-14 19:49:58.552984737 +0000 UTC m=+151.153288441"
Jul 14 19:50:03.876882 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:03.876846    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56db7c8bf4-9dr8h"]
Jul 14 19:50:03.905539 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:03.905510    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fbddf6585-p8j7l"]
Jul 14 19:50:03.907883 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:03.907862    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:50:03.920836 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:03.920813    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fbddf6585-p8j7l"]
Jul 14 19:50:03.954835 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:03.954801    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/882a6bd1-e610-47c2-a5bf-7f5ed64be422-trusted-ca-bundle\") pod \"console-7fbddf6585-p8j7l\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") " pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:50:03.954996 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:03.954893    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/882a6bd1-e610-47c2-a5bf-7f5ed64be422-console-config\") pod \"console-7fbddf6585-p8j7l\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") " pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:50:03.954996 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:03.954934    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/882a6bd1-e610-47c2-a5bf-7f5ed64be422-console-serving-cert\") pod \"console-7fbddf6585-p8j7l\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") " pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:50:03.954996 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:03.954978    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/882a6bd1-e610-47c2-a5bf-7f5ed64be422-console-oauth-config\") pod \"console-7fbddf6585-p8j7l\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") " pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:50:03.955124 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:03.955072    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5d7w\" (UniqueName: \"kubernetes.io/projected/882a6bd1-e610-47c2-a5bf-7f5ed64be422-kube-api-access-x5d7w\") pod \"console-7fbddf6585-p8j7l\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") " pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:50:03.955124 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:03.955101    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/882a6bd1-e610-47c2-a5bf-7f5ed64be422-service-ca\") pod \"console-7fbddf6585-p8j7l\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") " pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:50:03.955229 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:03.955123    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/882a6bd1-e610-47c2-a5bf-7f5ed64be422-oauth-serving-cert\") pod \"console-7fbddf6585-p8j7l\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") " pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:50:04.055867 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:04.055818    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/882a6bd1-e610-47c2-a5bf-7f5ed64be422-trusted-ca-bundle\") pod \"console-7fbddf6585-p8j7l\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") " pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:50:04.055867 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:04.055879    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/882a6bd1-e610-47c2-a5bf-7f5ed64be422-console-config\") pod \"console-7fbddf6585-p8j7l\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") " pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:50:04.056164 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:04.055908    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/882a6bd1-e610-47c2-a5bf-7f5ed64be422-console-serving-cert\") pod \"console-7fbddf6585-p8j7l\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") " pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:50:04.056164 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:04.055926    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/882a6bd1-e610-47c2-a5bf-7f5ed64be422-console-oauth-config\") pod \"console-7fbddf6585-p8j7l\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") " pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:50:04.056164 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:04.055965    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5d7w\" (UniqueName: \"kubernetes.io/projected/882a6bd1-e610-47c2-a5bf-7f5ed64be422-kube-api-access-x5d7w\") pod \"console-7fbddf6585-p8j7l\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") " pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:50:04.056164 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:04.055997    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/882a6bd1-e610-47c2-a5bf-7f5ed64be422-service-ca\") pod \"console-7fbddf6585-p8j7l\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") " pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:50:04.056164 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:04.056131    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/882a6bd1-e610-47c2-a5bf-7f5ed64be422-oauth-serving-cert\") pod \"console-7fbddf6585-p8j7l\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") " pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:50:04.056707 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:04.056684    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/882a6bd1-e610-47c2-a5bf-7f5ed64be422-console-config\") pod \"console-7fbddf6585-p8j7l\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") " pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:50:04.056804 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:04.056759    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/882a6bd1-e610-47c2-a5bf-7f5ed64be422-service-ca\") pod \"console-7fbddf6585-p8j7l\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") " pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:50:04.056868 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:04.056836    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/882a6bd1-e610-47c2-a5bf-7f5ed64be422-oauth-serving-cert\") pod \"console-7fbddf6585-p8j7l\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") " pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:50:04.056920 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:04.056888    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/882a6bd1-e610-47c2-a5bf-7f5ed64be422-trusted-ca-bundle\") pod \"console-7fbddf6585-p8j7l\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") " pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:50:04.058909 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:04.058891    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/882a6bd1-e610-47c2-a5bf-7f5ed64be422-console-serving-cert\") pod \"console-7fbddf6585-p8j7l\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") " pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:50:04.058963 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:04.058931    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/882a6bd1-e610-47c2-a5bf-7f5ed64be422-console-oauth-config\") pod \"console-7fbddf6585-p8j7l\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") " pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:50:04.065190 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:04.065147    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5d7w\" (UniqueName: \"kubernetes.io/projected/882a6bd1-e610-47c2-a5bf-7f5ed64be422-kube-api-access-x5d7w\") pod \"console-7fbddf6585-p8j7l\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") " pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:50:04.217210 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:04.217081    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:50:04.346615 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:04.346474    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fbddf6585-p8j7l"]
Jul 14 19:50:04.348836 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:50:04.348799    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod882a6bd1_e610_47c2_a5bf_7f5ed64be422.slice/crio-0ce18f04342068c116d3648d8af08ab39e831e9d7fd0511b4b66403273ead75e WatchSource:0}: Error finding container 0ce18f04342068c116d3648d8af08ab39e831e9d7fd0511b4b66403273ead75e: Status 404 returned error can't find the container with id 0ce18f04342068c116d3648d8af08ab39e831e9d7fd0511b4b66403273ead75e
Jul 14 19:50:04.558062 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:04.558022    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbddf6585-p8j7l" event={"ID":"882a6bd1-e610-47c2-a5bf-7f5ed64be422","Type":"ContainerStarted","Data":"08400d80f0200a429369e6d9141619c79f6ab02266d4fc0f9cf5e815707ab64f"}
Jul 14 19:50:04.558290 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:04.558087    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbddf6585-p8j7l" event={"ID":"882a6bd1-e610-47c2-a5bf-7f5ed64be422","Type":"ContainerStarted","Data":"0ce18f04342068c116d3648d8af08ab39e831e9d7fd0511b4b66403273ead75e"}
Jul 14 19:50:04.582737 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:04.582676    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7fbddf6585-p8j7l" podStartSLOduration=1.58265917 podStartE2EDuration="1.58265917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:50:04.581754488 +0000 UTC m=+157.182058192" watchObservedRunningTime="2026-07-14 19:50:04.58265917 +0000 UTC m=+157.182962872"
Jul 14 19:50:08.279870 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:08.279834    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56db7c8bf4-9dr8h"
Jul 14 19:50:14.217436 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:14.217398    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:50:14.217911 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:14.217465    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:50:14.221989 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:14.221958    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:50:14.599277 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:14.599250    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:50:14.647464 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:14.647436    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54d9f44c77-zd5gf"]
Jul 14 19:50:28.897846 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:28.897800    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56db7c8bf4-9dr8h" podUID="59a98800-4746-4bd4-bfcf-afc686b2dafa" containerName="console" containerID="cri-o://191fbc93ab17a2f0f4c1d0327f405326eb031d68a3d02fe8fdc80552af67a283" gracePeriod=15
Jul 14 19:50:29.140952 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.140927    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56db7c8bf4-9dr8h"
Jul 14 19:50:29.273234 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.273111    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkdbn\" (UniqueName: \"kubernetes.io/projected/59a98800-4746-4bd4-bfcf-afc686b2dafa-kube-api-access-qkdbn\") pod \"59a98800-4746-4bd4-bfcf-afc686b2dafa\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") "
Jul 14 19:50:29.273234 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.273208    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59a98800-4746-4bd4-bfcf-afc686b2dafa-console-oauth-config\") pod \"59a98800-4746-4bd4-bfcf-afc686b2dafa\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") "
Jul 14 19:50:29.273234 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.273227    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59a98800-4746-4bd4-bfcf-afc686b2dafa-console-serving-cert\") pod \"59a98800-4746-4bd4-bfcf-afc686b2dafa\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") "
Jul 14 19:50:29.273490 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.273333    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59a98800-4746-4bd4-bfcf-afc686b2dafa-console-config\") pod \"59a98800-4746-4bd4-bfcf-afc686b2dafa\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") "
Jul 14 19:50:29.273490 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.273419    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59a98800-4746-4bd4-bfcf-afc686b2dafa-trusted-ca-bundle\") pod \"59a98800-4746-4bd4-bfcf-afc686b2dafa\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") "
Jul 14 19:50:29.273490 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.273458    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59a98800-4746-4bd4-bfcf-afc686b2dafa-oauth-serving-cert\") pod \"59a98800-4746-4bd4-bfcf-afc686b2dafa\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") "
Jul 14 19:50:29.273638 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.273490    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59a98800-4746-4bd4-bfcf-afc686b2dafa-service-ca\") pod \"59a98800-4746-4bd4-bfcf-afc686b2dafa\" (UID: \"59a98800-4746-4bd4-bfcf-afc686b2dafa\") "
Jul 14 19:50:29.273811 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.273767    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a98800-4746-4bd4-bfcf-afc686b2dafa-console-config" (OuterVolumeSpecName: "console-config") pod "59a98800-4746-4bd4-bfcf-afc686b2dafa" (UID: "59a98800-4746-4bd4-bfcf-afc686b2dafa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:50:29.273811 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.273797    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a98800-4746-4bd4-bfcf-afc686b2dafa-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "59a98800-4746-4bd4-bfcf-afc686b2dafa" (UID: "59a98800-4746-4bd4-bfcf-afc686b2dafa"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:50:29.274165 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.273859    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a98800-4746-4bd4-bfcf-afc686b2dafa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "59a98800-4746-4bd4-bfcf-afc686b2dafa" (UID: "59a98800-4746-4bd4-bfcf-afc686b2dafa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:50:29.274165 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.274147    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a98800-4746-4bd4-bfcf-afc686b2dafa-service-ca" (OuterVolumeSpecName: "service-ca") pod "59a98800-4746-4bd4-bfcf-afc686b2dafa" (UID: "59a98800-4746-4bd4-bfcf-afc686b2dafa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:50:29.275550 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.275528    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a98800-4746-4bd4-bfcf-afc686b2dafa-kube-api-access-qkdbn" (OuterVolumeSpecName: "kube-api-access-qkdbn") pod "59a98800-4746-4bd4-bfcf-afc686b2dafa" (UID: "59a98800-4746-4bd4-bfcf-afc686b2dafa"). InnerVolumeSpecName "kube-api-access-qkdbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:50:29.275659 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.275574    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a98800-4746-4bd4-bfcf-afc686b2dafa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "59a98800-4746-4bd4-bfcf-afc686b2dafa" (UID: "59a98800-4746-4bd4-bfcf-afc686b2dafa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:50:29.275881 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.275855    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a98800-4746-4bd4-bfcf-afc686b2dafa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "59a98800-4746-4bd4-bfcf-afc686b2dafa" (UID: "59a98800-4746-4bd4-bfcf-afc686b2dafa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:50:29.374460 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.374423    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59a98800-4746-4bd4-bfcf-afc686b2dafa-trusted-ca-bundle\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:50:29.374460 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.374455    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59a98800-4746-4bd4-bfcf-afc686b2dafa-oauth-serving-cert\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:50:29.374460 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.374465    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59a98800-4746-4bd4-bfcf-afc686b2dafa-service-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:50:29.374674 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.374476    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkdbn\" (UniqueName: \"kubernetes.io/projected/59a98800-4746-4bd4-bfcf-afc686b2dafa-kube-api-access-qkdbn\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:50:29.374674 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.374486    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59a98800-4746-4bd4-bfcf-afc686b2dafa-console-oauth-config\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:50:29.374674 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.374511    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59a98800-4746-4bd4-bfcf-afc686b2dafa-console-serving-cert\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:50:29.374674 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.374520    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59a98800-4746-4bd4-bfcf-afc686b2dafa-console-config\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:50:29.652388 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.652351    2560 generic.go:356] "Generic (PLEG): container finished" podID="59a98800-4746-4bd4-bfcf-afc686b2dafa" containerID="191fbc93ab17a2f0f4c1d0327f405326eb031d68a3d02fe8fdc80552af67a283" exitCode=0
Jul 14 19:50:29.652570 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.652407    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56db7c8bf4-9dr8h"
Jul 14 19:50:29.652570 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.652436    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56db7c8bf4-9dr8h" event={"ID":"59a98800-4746-4bd4-bfcf-afc686b2dafa","Type":"ContainerDied","Data":"191fbc93ab17a2f0f4c1d0327f405326eb031d68a3d02fe8fdc80552af67a283"}
Jul 14 19:50:29.652570 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.652486    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56db7c8bf4-9dr8h" event={"ID":"59a98800-4746-4bd4-bfcf-afc686b2dafa","Type":"ContainerDied","Data":"7c947459fd2549cb3044b1c538e9e856f3f064e07e77ab4e9c02e3252e3ae25d"}
Jul 14 19:50:29.652570 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.652506    2560 scope.go:117] "RemoveContainer" containerID="191fbc93ab17a2f0f4c1d0327f405326eb031d68a3d02fe8fdc80552af67a283"
Jul 14 19:50:29.661658 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.661635    2560 scope.go:117] "RemoveContainer" containerID="191fbc93ab17a2f0f4c1d0327f405326eb031d68a3d02fe8fdc80552af67a283"
Jul 14 19:50:29.661940 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:50:29.661923    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"191fbc93ab17a2f0f4c1d0327f405326eb031d68a3d02fe8fdc80552af67a283\": container with ID starting with 191fbc93ab17a2f0f4c1d0327f405326eb031d68a3d02fe8fdc80552af67a283 not found: ID does not exist" containerID="191fbc93ab17a2f0f4c1d0327f405326eb031d68a3d02fe8fdc80552af67a283"
Jul 14 19:50:29.661992 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.661949    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"191fbc93ab17a2f0f4c1d0327f405326eb031d68a3d02fe8fdc80552af67a283"} err="failed to get container status \"191fbc93ab17a2f0f4c1d0327f405326eb031d68a3d02fe8fdc80552af67a283\": rpc error: code = NotFound desc = could not find container \"191fbc93ab17a2f0f4c1d0327f405326eb031d68a3d02fe8fdc80552af67a283\": container with ID starting with 191fbc93ab17a2f0f4c1d0327f405326eb031d68a3d02fe8fdc80552af67a283 not found: ID does not exist"
Jul 14 19:50:29.680243 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.680222    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56db7c8bf4-9dr8h"]
Jul 14 19:50:29.684727 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.684707    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56db7c8bf4-9dr8h"]
Jul 14 19:50:29.888056 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:29.888025    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59a98800-4746-4bd4-bfcf-afc686b2dafa" path="/var/lib/kubelet/pods/59a98800-4746-4bd4-bfcf-afc686b2dafa/volumes"
Jul 14 19:50:39.667464 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:39.667421    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54d9f44c77-zd5gf" podUID="30180731-6603-4f6f-96fa-b9b78001bb6a" containerName="console" containerID="cri-o://1f47e076a34759ee69f1155c4753ba471e5f9e2e90cf0493244ed6640255a02b" gracePeriod=15
Jul 14 19:50:39.913168 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:39.913141    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:50:40.062155 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.062117    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30180731-6603-4f6f-96fa-b9b78001bb6a-trusted-ca-bundle\") pod \"30180731-6603-4f6f-96fa-b9b78001bb6a\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") "
Jul 14 19:50:40.062370 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.062196    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30180731-6603-4f6f-96fa-b9b78001bb6a-console-serving-cert\") pod \"30180731-6603-4f6f-96fa-b9b78001bb6a\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") "
Jul 14 19:50:40.062370 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.062231    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30180731-6603-4f6f-96fa-b9b78001bb6a-console-oauth-config\") pod \"30180731-6603-4f6f-96fa-b9b78001bb6a\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") "
Jul 14 19:50:40.062370 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.062300    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30180731-6603-4f6f-96fa-b9b78001bb6a-console-config\") pod \"30180731-6603-4f6f-96fa-b9b78001bb6a\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") "
Jul 14 19:50:40.062533 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.062503    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30180731-6603-4f6f-96fa-b9b78001bb6a-oauth-serving-cert\") pod \"30180731-6603-4f6f-96fa-b9b78001bb6a\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") "
Jul 14 19:50:40.062592 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.062538    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30180731-6603-4f6f-96fa-b9b78001bb6a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "30180731-6603-4f6f-96fa-b9b78001bb6a" (UID: "30180731-6603-4f6f-96fa-b9b78001bb6a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:50:40.062592 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.062560    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30180731-6603-4f6f-96fa-b9b78001bb6a-service-ca\") pod \"30180731-6603-4f6f-96fa-b9b78001bb6a\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") "
Jul 14 19:50:40.062695 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.062626    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzkfg\" (UniqueName: \"kubernetes.io/projected/30180731-6603-4f6f-96fa-b9b78001bb6a-kube-api-access-dzkfg\") pod \"30180731-6603-4f6f-96fa-b9b78001bb6a\" (UID: \"30180731-6603-4f6f-96fa-b9b78001bb6a\") "
Jul 14 19:50:40.062751 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.062683    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30180731-6603-4f6f-96fa-b9b78001bb6a-console-config" (OuterVolumeSpecName: "console-config") pod "30180731-6603-4f6f-96fa-b9b78001bb6a" (UID: "30180731-6603-4f6f-96fa-b9b78001bb6a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:50:40.062856 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.062838    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30180731-6603-4f6f-96fa-b9b78001bb6a-console-config\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:50:40.062916 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.062862    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30180731-6603-4f6f-96fa-b9b78001bb6a-trusted-ca-bundle\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:50:40.062916 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.062882    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30180731-6603-4f6f-96fa-b9b78001bb6a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "30180731-6603-4f6f-96fa-b9b78001bb6a" (UID: "30180731-6603-4f6f-96fa-b9b78001bb6a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:50:40.063023 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.062921    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30180731-6603-4f6f-96fa-b9b78001bb6a-service-ca" (OuterVolumeSpecName: "service-ca") pod "30180731-6603-4f6f-96fa-b9b78001bb6a" (UID: "30180731-6603-4f6f-96fa-b9b78001bb6a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:50:40.064972 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.064946    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30180731-6603-4f6f-96fa-b9b78001bb6a-kube-api-access-dzkfg" (OuterVolumeSpecName: "kube-api-access-dzkfg") pod "30180731-6603-4f6f-96fa-b9b78001bb6a" (UID: "30180731-6603-4f6f-96fa-b9b78001bb6a"). InnerVolumeSpecName "kube-api-access-dzkfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:50:40.065083 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.065065    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30180731-6603-4f6f-96fa-b9b78001bb6a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "30180731-6603-4f6f-96fa-b9b78001bb6a" (UID: "30180731-6603-4f6f-96fa-b9b78001bb6a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:50:40.065185 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.065146    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30180731-6603-4f6f-96fa-b9b78001bb6a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "30180731-6603-4f6f-96fa-b9b78001bb6a" (UID: "30180731-6603-4f6f-96fa-b9b78001bb6a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:50:40.163639 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.163585    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30180731-6603-4f6f-96fa-b9b78001bb6a-console-oauth-config\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:50:40.163639 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.163632    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30180731-6603-4f6f-96fa-b9b78001bb6a-oauth-serving-cert\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:50:40.163857 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.163649    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30180731-6603-4f6f-96fa-b9b78001bb6a-service-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:50:40.163857 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.163668    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzkfg\" (UniqueName: \"kubernetes.io/projected/30180731-6603-4f6f-96fa-b9b78001bb6a-kube-api-access-dzkfg\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:50:40.163857 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.163678    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30180731-6603-4f6f-96fa-b9b78001bb6a-console-serving-cert\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:50:40.694504 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.694467    2560 generic.go:356] "Generic (PLEG): container finished" podID="30180731-6603-4f6f-96fa-b9b78001bb6a" containerID="1f47e076a34759ee69f1155c4753ba471e5f9e2e90cf0493244ed6640255a02b" exitCode=0
Jul 14 19:50:40.694947 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.694513    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d9f44c77-zd5gf"
Jul 14 19:50:40.694947 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.694515    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d9f44c77-zd5gf" event={"ID":"30180731-6603-4f6f-96fa-b9b78001bb6a","Type":"ContainerDied","Data":"1f47e076a34759ee69f1155c4753ba471e5f9e2e90cf0493244ed6640255a02b"}
Jul 14 19:50:40.694947 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.694667    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d9f44c77-zd5gf" event={"ID":"30180731-6603-4f6f-96fa-b9b78001bb6a","Type":"ContainerDied","Data":"3fd8ae7b1569383301b4cabf5999527f5e2cbacf4e5a71c310b41df80b8fcb76"}
Jul 14 19:50:40.694947 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.694705    2560 scope.go:117] "RemoveContainer" containerID="1f47e076a34759ee69f1155c4753ba471e5f9e2e90cf0493244ed6640255a02b"
Jul 14 19:50:40.703143 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.703126    2560 scope.go:117] "RemoveContainer" containerID="1f47e076a34759ee69f1155c4753ba471e5f9e2e90cf0493244ed6640255a02b"
Jul 14 19:50:40.703405 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:50:40.703386    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f47e076a34759ee69f1155c4753ba471e5f9e2e90cf0493244ed6640255a02b\": container with ID starting with 1f47e076a34759ee69f1155c4753ba471e5f9e2e90cf0493244ed6640255a02b not found: ID does not exist" containerID="1f47e076a34759ee69f1155c4753ba471e5f9e2e90cf0493244ed6640255a02b"
Jul 14 19:50:40.703453 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.703413    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f47e076a34759ee69f1155c4753ba471e5f9e2e90cf0493244ed6640255a02b"} err="failed to get container status \"1f47e076a34759ee69f1155c4753ba471e5f9e2e90cf0493244ed6640255a02b\": rpc error: code = NotFound desc = could not find container \"1f47e076a34759ee69f1155c4753ba471e5f9e2e90cf0493244ed6640255a02b\": container with ID starting with 1f47e076a34759ee69f1155c4753ba471e5f9e2e90cf0493244ed6640255a02b not found: ID does not exist"
Jul 14 19:50:40.717919 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.717893    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54d9f44c77-zd5gf"]
Jul 14 19:50:40.723038 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:40.723018    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54d9f44c77-zd5gf"]
Jul 14 19:50:41.887997 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:50:41.887963    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30180731-6603-4f6f-96fa-b9b78001bb6a" path="/var/lib/kubelet/pods/30180731-6603-4f6f-96fa-b9b78001bb6a/volumes"
Jul 14 19:51:48.763980 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:51:48.763903    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rpjff"]
Jul 14 19:51:48.764472 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:51:48.764202    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30180731-6603-4f6f-96fa-b9b78001bb6a" containerName="console"
Jul 14 19:51:48.764472 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:51:48.764213    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="30180731-6603-4f6f-96fa-b9b78001bb6a" containerName="console"
Jul 14 19:51:48.764472 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:51:48.764225    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59a98800-4746-4bd4-bfcf-afc686b2dafa" containerName="console"
Jul 14 19:51:48.764472 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:51:48.764232    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="59a98800-4746-4bd4-bfcf-afc686b2dafa" containerName="console"
Jul 14 19:51:48.764472 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:51:48.764284    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="59a98800-4746-4bd4-bfcf-afc686b2dafa" containerName="console"
Jul 14 19:51:48.764472 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:51:48.764293    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="30180731-6603-4f6f-96fa-b9b78001bb6a" containerName="console"
Jul 14 19:51:48.766649 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:51:48.766632    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-rpjff"
Jul 14 19:51:48.769035 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:51:48.769015    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 14 19:51:48.769951 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:51:48.769936    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-mtdfz\""
Jul 14 19:51:48.770001 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:51:48.769940    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 14 19:51:48.774774 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:51:48.774754    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rpjff"]
Jul 14 19:51:48.923506 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:51:48.923469    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndxzw\" (UniqueName: \"kubernetes.io/projected/8e6d3c97-343a-45a1-8d70-37a21116248f-kube-api-access-ndxzw\") pod \"cert-manager-55b4d954d5-rpjff\" (UID: \"8e6d3c97-343a-45a1-8d70-37a21116248f\") " pod="cert-manager/cert-manager-55b4d954d5-rpjff"
Jul 14 19:51:48.923691 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:51:48.923536    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8e6d3c97-343a-45a1-8d70-37a21116248f-bound-sa-token\") pod \"cert-manager-55b4d954d5-rpjff\" (UID: \"8e6d3c97-343a-45a1-8d70-37a21116248f\") " pod="cert-manager/cert-manager-55b4d954d5-rpjff"
Jul 14 19:51:49.024826 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:51:49.024739    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndxzw\" (UniqueName: \"kubernetes.io/projected/8e6d3c97-343a-45a1-8d70-37a21116248f-kube-api-access-ndxzw\") pod \"cert-manager-55b4d954d5-rpjff\" (UID: \"8e6d3c97-343a-45a1-8d70-37a21116248f\") " pod="cert-manager/cert-manager-55b4d954d5-rpjff"
Jul 14 19:51:49.024985 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:51:49.024874    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8e6d3c97-343a-45a1-8d70-37a21116248f-bound-sa-token\") pod \"cert-manager-55b4d954d5-rpjff\" (UID: \"8e6d3c97-343a-45a1-8d70-37a21116248f\") " pod="cert-manager/cert-manager-55b4d954d5-rpjff"
Jul 14 19:51:49.034235 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:51:49.034206    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndxzw\" (UniqueName: \"kubernetes.io/projected/8e6d3c97-343a-45a1-8d70-37a21116248f-kube-api-access-ndxzw\") pod \"cert-manager-55b4d954d5-rpjff\" (UID: \"8e6d3c97-343a-45a1-8d70-37a21116248f\") " pod="cert-manager/cert-manager-55b4d954d5-rpjff"
Jul 14 19:51:49.034235 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:51:49.034216    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8e6d3c97-343a-45a1-8d70-37a21116248f-bound-sa-token\") pod \"cert-manager-55b4d954d5-rpjff\" (UID: \"8e6d3c97-343a-45a1-8d70-37a21116248f\") " pod="cert-manager/cert-manager-55b4d954d5-rpjff"
Jul 14 19:51:49.090266 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:51:49.090231    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-rpjff"
Jul 14 19:51:49.220148 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:51:49.220042    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rpjff"]
Jul 14 19:51:49.222833 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:51:49.222803    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e6d3c97_343a_45a1_8d70_37a21116248f.slice/crio-8bd671aaade37a1d79270618bc657213f0e05c80e3fc2cfcc626f726314c4668 WatchSource:0}: Error finding container 8bd671aaade37a1d79270618bc657213f0e05c80e3fc2cfcc626f726314c4668: Status 404 returned error can't find the container with id 8bd671aaade37a1d79270618bc657213f0e05c80e3fc2cfcc626f726314c4668
Jul 14 19:51:49.964698 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:51:49.964657    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-rpjff" event={"ID":"8e6d3c97-343a-45a1-8d70-37a21116248f","Type":"ContainerStarted","Data":"8bd671aaade37a1d79270618bc657213f0e05c80e3fc2cfcc626f726314c4668"}
Jul 14 19:51:52.977984 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:51:52.977950    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-rpjff" event={"ID":"8e6d3c97-343a-45a1-8d70-37a21116248f","Type":"ContainerStarted","Data":"67779ad75ad689bcb9ac5806490dc5a8c4c07648668d49de29216107043962b9"}
Jul 14 19:51:52.993386 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:51:52.993292    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-rpjff" podStartSLOduration=2.124122444 podStartE2EDuration="4.993273557s" totalImagesPullingTime="2.869151113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:51:52.993012729 +0000 UTC m=+265.593316444" watchObservedRunningTime="2026-07-14 19:51:52.993273557 +0000 UTC m=+265.593577264"
Jul 14 19:52:19.355749 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:19.355713    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c597f769-dckpj"]
Jul 14 19:52:19.359203 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:19.359160    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-dckpj"
Jul 14 19:52:19.364201 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:19.364165    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 14 19:52:19.364433 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:19.364416    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mfbvv\""
Jul 14 19:52:19.364617 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:19.364599    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 14 19:52:19.364745 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:19.364638    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 14 19:52:19.366300 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:19.366285    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 14 19:52:19.377440 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:19.377411    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c597f769-dckpj"]
Jul 14 19:52:19.488698 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:19.488662    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8a4667e6-2144-4b5f-9b6a-4e07d9b75350-manifests\") pod \"opendatahub-operator-controller-manager-6c597f769-dckpj\" (UID: \"8a4667e6-2144-4b5f-9b6a-4e07d9b75350\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-dckpj"
Jul 14 19:52:19.488698 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:19.488700    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a4667e6-2144-4b5f-9b6a-4e07d9b75350-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c597f769-dckpj\" (UID: \"8a4667e6-2144-4b5f-9b6a-4e07d9b75350\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-dckpj"
Jul 14 19:52:19.488908 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:19.488731    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a4667e6-2144-4b5f-9b6a-4e07d9b75350-tmp\") pod \"opendatahub-operator-controller-manager-6c597f769-dckpj\" (UID: \"8a4667e6-2144-4b5f-9b6a-4e07d9b75350\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-dckpj"
Jul 14 19:52:19.488908 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:19.488807    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2g87\" (UniqueName: \"kubernetes.io/projected/8a4667e6-2144-4b5f-9b6a-4e07d9b75350-kube-api-access-v2g87\") pod \"opendatahub-operator-controller-manager-6c597f769-dckpj\" (UID: \"8a4667e6-2144-4b5f-9b6a-4e07d9b75350\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-dckpj"
Jul 14 19:52:19.488908 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:19.488853    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8a4667e6-2144-4b5f-9b6a-4e07d9b75350-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c597f769-dckpj\" (UID: \"8a4667e6-2144-4b5f-9b6a-4e07d9b75350\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-dckpj"
Jul 14 19:52:19.589405 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:19.589367    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8a4667e6-2144-4b5f-9b6a-4e07d9b75350-manifests\") pod \"opendatahub-operator-controller-manager-6c597f769-dckpj\" (UID: \"8a4667e6-2144-4b5f-9b6a-4e07d9b75350\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-dckpj"
Jul 14 19:52:19.589405 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:19.589409    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a4667e6-2144-4b5f-9b6a-4e07d9b75350-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c597f769-dckpj\" (UID: \"8a4667e6-2144-4b5f-9b6a-4e07d9b75350\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-dckpj"
Jul 14 19:52:19.589629 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:19.589449    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a4667e6-2144-4b5f-9b6a-4e07d9b75350-tmp\") pod \"opendatahub-operator-controller-manager-6c597f769-dckpj\" (UID: \"8a4667e6-2144-4b5f-9b6a-4e07d9b75350\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-dckpj"
Jul 14 19:52:19.589629 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:19.589490    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2g87\" (UniqueName: \"kubernetes.io/projected/8a4667e6-2144-4b5f-9b6a-4e07d9b75350-kube-api-access-v2g87\") pod \"opendatahub-operator-controller-manager-6c597f769-dckpj\" (UID: \"8a4667e6-2144-4b5f-9b6a-4e07d9b75350\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-dckpj"
Jul 14 19:52:19.589629 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:19.589541    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8a4667e6-2144-4b5f-9b6a-4e07d9b75350-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c597f769-dckpj\" (UID: \"8a4667e6-2144-4b5f-9b6a-4e07d9b75350\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-dckpj"
Jul 14 19:52:19.589784 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:19.589775    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8a4667e6-2144-4b5f-9b6a-4e07d9b75350-manifests\") pod \"opendatahub-operator-controller-manager-6c597f769-dckpj\" (UID: \"8a4667e6-2144-4b5f-9b6a-4e07d9b75350\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-dckpj"
Jul 14 19:52:19.591887 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:19.591856    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a4667e6-2144-4b5f-9b6a-4e07d9b75350-tmp\") pod \"opendatahub-operator-controller-manager-6c597f769-dckpj\" (UID: \"8a4667e6-2144-4b5f-9b6a-4e07d9b75350\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-dckpj"
Jul 14 19:52:19.592159 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:19.592143    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8a4667e6-2144-4b5f-9b6a-4e07d9b75350-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c597f769-dckpj\" (UID: \"8a4667e6-2144-4b5f-9b6a-4e07d9b75350\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-dckpj"
Jul 14 19:52:19.592221 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:19.592187    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a4667e6-2144-4b5f-9b6a-4e07d9b75350-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c597f769-dckpj\" (UID: \"8a4667e6-2144-4b5f-9b6a-4e07d9b75350\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-dckpj"
Jul 14 19:52:19.599947 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:19.599927    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2g87\" (UniqueName: \"kubernetes.io/projected/8a4667e6-2144-4b5f-9b6a-4e07d9b75350-kube-api-access-v2g87\") pod \"opendatahub-operator-controller-manager-6c597f769-dckpj\" (UID: \"8a4667e6-2144-4b5f-9b6a-4e07d9b75350\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-dckpj"
Jul 14 19:52:19.670021 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:19.669923    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-dckpj"
Jul 14 19:52:19.806392 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:19.806370    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c597f769-dckpj"]
Jul 14 19:52:19.808773 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:52:19.808742    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a4667e6_2144_4b5f_9b6a_4e07d9b75350.slice/crio-f004ec8f899295a0e2776a3b0b891449e296bfe2bc7d9cf917f8eb5bdb740863 WatchSource:0}: Error finding container f004ec8f899295a0e2776a3b0b891449e296bfe2bc7d9cf917f8eb5bdb740863: Status 404 returned error can't find the container with id f004ec8f899295a0e2776a3b0b891449e296bfe2bc7d9cf917f8eb5bdb740863
Jul 14 19:52:20.092902 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:20.092868    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-dckpj" event={"ID":"8a4667e6-2144-4b5f-9b6a-4e07d9b75350","Type":"ContainerStarted","Data":"f004ec8f899295a0e2776a3b0b891449e296bfe2bc7d9cf917f8eb5bdb740863"}
Jul 14 19:52:23.107143 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:23.107063    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-dckpj" event={"ID":"8a4667e6-2144-4b5f-9b6a-4e07d9b75350","Type":"ContainerStarted","Data":"610645472dc2c9e5070b5b97c49e05d965f8e107a094f08c1d94d6ca75640e1b"}
Jul 14 19:52:24.112853 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:24.112815    2560 generic.go:356] "Generic (PLEG): container finished" podID="8a4667e6-2144-4b5f-9b6a-4e07d9b75350" containerID="610645472dc2c9e5070b5b97c49e05d965f8e107a094f08c1d94d6ca75640e1b" exitCode=0
Jul 14 19:52:24.113325 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:24.112889    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-dckpj" event={"ID":"8a4667e6-2144-4b5f-9b6a-4e07d9b75350","Type":"ContainerDied","Data":"610645472dc2c9e5070b5b97c49e05d965f8e107a094f08c1d94d6ca75640e1b"}
Jul 14 19:52:25.120191 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:25.120133    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-dckpj" event={"ID":"8a4667e6-2144-4b5f-9b6a-4e07d9b75350","Type":"ContainerStarted","Data":"a1087b76122570e4c821dd77561cb3de30203e435d53c0dec6312e7f64aeda2b"}
Jul 14 19:52:25.120690 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:25.120279    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-dckpj"
Jul 14 19:52:25.144363 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:25.144307    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-dckpj" podStartSLOduration=1.729527473 podStartE2EDuration="6.144289196s" totalImagesPullingTime="3.384360612s" totalInitContainerRuntime="1.030401111s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:52:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:52:25.143493612 +0000 UTC m=+297.743797327" watchObservedRunningTime="2026-07-14 19:52:25.144289196 +0000 UTC m=+297.744592898"
Jul 14 19:52:27.788609 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:27.788581    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tzr6_4a36af4c-9eac-465f-9a0e-d65a382530f7/ovn-acl-logging/0.log"
Jul 14 19:52:27.789421 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:27.789400    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tzr6_4a36af4c-9eac-465f-9a0e-d65a382530f7/ovn-acl-logging/0.log"
Jul 14 19:52:27.795477 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:27.795457    2560 kubelet.go:1648] "Image garbage collection succeeded"
Jul 14 19:52:36.126809 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:36.126780    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-dckpj"
Jul 14 19:52:39.477730 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:39.477699    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-jfck4"]
Jul 14 19:52:39.480309 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:39.480293    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-jfck4"
Jul 14 19:52:39.482802 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:39.482785    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 14 19:52:39.502833 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:39.502796    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-jfck4"]
Jul 14 19:52:39.658760 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:39.658727    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d266cde7-b5bd-4ae6-8977-be6bed9824aa-tls-certs\") pod \"model-serving-api-54d89bd5fc-jfck4\" (UID: \"d266cde7-b5bd-4ae6-8977-be6bed9824aa\") " pod="opendatahub/model-serving-api-54d89bd5fc-jfck4"
Jul 14 19:52:39.658927 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:39.658762    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xgc6\" (UniqueName: \"kubernetes.io/projected/d266cde7-b5bd-4ae6-8977-be6bed9824aa-kube-api-access-9xgc6\") pod \"model-serving-api-54d89bd5fc-jfck4\" (UID: \"d266cde7-b5bd-4ae6-8977-be6bed9824aa\") " pod="opendatahub/model-serving-api-54d89bd5fc-jfck4"
Jul 14 19:52:39.759444 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:39.759350    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d266cde7-b5bd-4ae6-8977-be6bed9824aa-tls-certs\") pod \"model-serving-api-54d89bd5fc-jfck4\" (UID: \"d266cde7-b5bd-4ae6-8977-be6bed9824aa\") " pod="opendatahub/model-serving-api-54d89bd5fc-jfck4"
Jul 14 19:52:39.759444 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:39.759392    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xgc6\" (UniqueName: \"kubernetes.io/projected/d266cde7-b5bd-4ae6-8977-be6bed9824aa-kube-api-access-9xgc6\") pod \"model-serving-api-54d89bd5fc-jfck4\" (UID: \"d266cde7-b5bd-4ae6-8977-be6bed9824aa\") " pod="opendatahub/model-serving-api-54d89bd5fc-jfck4"
Jul 14 19:52:39.759658 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:52:39.759500    2560 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 14 19:52:39.759658 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:52:39.759561    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d266cde7-b5bd-4ae6-8977-be6bed9824aa-tls-certs podName:d266cde7-b5bd-4ae6-8977-be6bed9824aa nodeName:}" failed. No retries permitted until 2026-07-14 19:52:40.259545236 +0000 UTC m=+312.859848917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d266cde7-b5bd-4ae6-8977-be6bed9824aa-tls-certs") pod "model-serving-api-54d89bd5fc-jfck4" (UID: "d266cde7-b5bd-4ae6-8977-be6bed9824aa") : secret "model-serving-api-tls" not found
Jul 14 19:52:39.769365 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:39.769340    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xgc6\" (UniqueName: \"kubernetes.io/projected/d266cde7-b5bd-4ae6-8977-be6bed9824aa-kube-api-access-9xgc6\") pod \"model-serving-api-54d89bd5fc-jfck4\" (UID: \"d266cde7-b5bd-4ae6-8977-be6bed9824aa\") " pod="opendatahub/model-serving-api-54d89bd5fc-jfck4"
Jul 14 19:52:40.264625 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:40.264585    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d266cde7-b5bd-4ae6-8977-be6bed9824aa-tls-certs\") pod \"model-serving-api-54d89bd5fc-jfck4\" (UID: \"d266cde7-b5bd-4ae6-8977-be6bed9824aa\") " pod="opendatahub/model-serving-api-54d89bd5fc-jfck4"
Jul 14 19:52:40.267147 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:40.267120    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d266cde7-b5bd-4ae6-8977-be6bed9824aa-tls-certs\") pod \"model-serving-api-54d89bd5fc-jfck4\" (UID: \"d266cde7-b5bd-4ae6-8977-be6bed9824aa\") " pod="opendatahub/model-serving-api-54d89bd5fc-jfck4"
Jul 14 19:52:40.390183 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:40.390135    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-jfck4"
Jul 14 19:52:40.525033 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:40.524949    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-jfck4"]
Jul 14 19:52:40.528561 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:52:40.528524    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd266cde7_b5bd_4ae6_8977_be6bed9824aa.slice/crio-1ad76aa61003a6d7b86b6ba7df0a897da0e442f9352fcb759bbd89990bf1f2a1 WatchSource:0}: Error finding container 1ad76aa61003a6d7b86b6ba7df0a897da0e442f9352fcb759bbd89990bf1f2a1: Status 404 returned error can't find the container with id 1ad76aa61003a6d7b86b6ba7df0a897da0e442f9352fcb759bbd89990bf1f2a1
Jul 14 19:52:40.530641 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:40.530626    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 19:52:41.182341 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:41.182306    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-jfck4" event={"ID":"d266cde7-b5bd-4ae6-8977-be6bed9824aa","Type":"ContainerStarted","Data":"1ad76aa61003a6d7b86b6ba7df0a897da0e442f9352fcb759bbd89990bf1f2a1"}
Jul 14 19:52:43.191784 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:43.191729    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-jfck4" event={"ID":"d266cde7-b5bd-4ae6-8977-be6bed9824aa","Type":"ContainerStarted","Data":"a004c4abf8ee68151bfe7651b199ae127f36259e48b0bcd4947ebd0a9719efc3"}
Jul 14 19:52:43.192196 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:43.191890    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-jfck4"
Jul 14 19:52:43.209773 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:43.209696    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-jfck4" podStartSLOduration=2.224839217 podStartE2EDuration="4.209675776s" totalImagesPullingTime="1.984836559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:52:43.207862174 +0000 UTC m=+315.808165879" watchObservedRunningTime="2026-07-14 19:52:43.209675776 +0000 UTC m=+315.809979481"
Jul 14 19:52:46.592588 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:46.592555    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-77d6d59f96-69z6h"]
Jul 14 19:52:46.595812 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:46.595775    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-69z6h"
Jul 14 19:52:46.599889 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:46.599862    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 14 19:52:46.600006 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:46.599862    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 14 19:52:46.600006 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:46.599927    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 14 19:52:46.600006 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:46.599862    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-kvspf\""
Jul 14 19:52:46.600238 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:46.600223    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 14 19:52:46.600318 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:46.600250    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 14 19:52:46.608653 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:46.608633    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-77d6d59f96-69z6h"]
Jul 14 19:52:46.617468 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:46.617445    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmf54\" (UniqueName: \"kubernetes.io/projected/411bcb53-f540-49a8-8665-3b257034d8f4-kube-api-access-vmf54\") pod \"lws-controller-manager-77d6d59f96-69z6h\" (UID: \"411bcb53-f540-49a8-8665-3b257034d8f4\") " pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-69z6h"
Jul 14 19:52:46.617601 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:46.617480    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/411bcb53-f540-49a8-8665-3b257034d8f4-manager-config\") pod \"lws-controller-manager-77d6d59f96-69z6h\" (UID: \"411bcb53-f540-49a8-8665-3b257034d8f4\") " pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-69z6h"
Jul 14 19:52:46.617601 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:46.617558    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/411bcb53-f540-49a8-8665-3b257034d8f4-cert\") pod \"lws-controller-manager-77d6d59f96-69z6h\" (UID: \"411bcb53-f540-49a8-8665-3b257034d8f4\") " pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-69z6h"
Jul 14 19:52:46.617727 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:46.617603    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/411bcb53-f540-49a8-8665-3b257034d8f4-metrics-cert\") pod \"lws-controller-manager-77d6d59f96-69z6h\" (UID: \"411bcb53-f540-49a8-8665-3b257034d8f4\") " pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-69z6h"
Jul 14 19:52:46.718940 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:46.718896    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/411bcb53-f540-49a8-8665-3b257034d8f4-metrics-cert\") pod \"lws-controller-manager-77d6d59f96-69z6h\" (UID: \"411bcb53-f540-49a8-8665-3b257034d8f4\") " pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-69z6h"
Jul 14 19:52:46.719112 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:46.719016    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmf54\" (UniqueName: \"kubernetes.io/projected/411bcb53-f540-49a8-8665-3b257034d8f4-kube-api-access-vmf54\") pod \"lws-controller-manager-77d6d59f96-69z6h\" (UID: \"411bcb53-f540-49a8-8665-3b257034d8f4\") " pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-69z6h"
Jul 14 19:52:46.719112 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:46.719051    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/411bcb53-f540-49a8-8665-3b257034d8f4-manager-config\") pod \"lws-controller-manager-77d6d59f96-69z6h\" (UID: \"411bcb53-f540-49a8-8665-3b257034d8f4\") " pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-69z6h"
Jul 14 19:52:46.719220 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:46.719136    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/411bcb53-f540-49a8-8665-3b257034d8f4-cert\") pod \"lws-controller-manager-77d6d59f96-69z6h\" (UID: \"411bcb53-f540-49a8-8665-3b257034d8f4\") " pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-69z6h"
Jul 14 19:52:46.720133 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:46.720097    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/411bcb53-f540-49a8-8665-3b257034d8f4-manager-config\") pod \"lws-controller-manager-77d6d59f96-69z6h\" (UID: \"411bcb53-f540-49a8-8665-3b257034d8f4\") " pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-69z6h"
Jul 14 19:52:46.721706 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:46.721683    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/411bcb53-f540-49a8-8665-3b257034d8f4-cert\") pod \"lws-controller-manager-77d6d59f96-69z6h\" (UID: \"411bcb53-f540-49a8-8665-3b257034d8f4\") " pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-69z6h"
Jul 14 19:52:46.721817 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:46.721714    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/411bcb53-f540-49a8-8665-3b257034d8f4-metrics-cert\") pod \"lws-controller-manager-77d6d59f96-69z6h\" (UID: \"411bcb53-f540-49a8-8665-3b257034d8f4\") " pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-69z6h"
Jul 14 19:52:46.728323 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:46.728302    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmf54\" (UniqueName: \"kubernetes.io/projected/411bcb53-f540-49a8-8665-3b257034d8f4-kube-api-access-vmf54\") pod \"lws-controller-manager-77d6d59f96-69z6h\" (UID: \"411bcb53-f540-49a8-8665-3b257034d8f4\") " pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-69z6h"
Jul 14 19:52:46.905477 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:46.905387    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-69z6h"
Jul 14 19:52:47.035859 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:47.035828    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-77d6d59f96-69z6h"]
Jul 14 19:52:47.039215 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:52:47.039190    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod411bcb53_f540_49a8_8665_3b257034d8f4.slice/crio-72dad6abf9b38e247d368d93743f9a8193810244fc55e19aaa68242648900b14 WatchSource:0}: Error finding container 72dad6abf9b38e247d368d93743f9a8193810244fc55e19aaa68242648900b14: Status 404 returned error can't find the container with id 72dad6abf9b38e247d368d93743f9a8193810244fc55e19aaa68242648900b14
Jul 14 19:52:47.208339 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:47.208248    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-69z6h" event={"ID":"411bcb53-f540-49a8-8665-3b257034d8f4","Type":"ContainerStarted","Data":"72dad6abf9b38e247d368d93743f9a8193810244fc55e19aaa68242648900b14"}
Jul 14 19:52:48.659661 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:48.659619    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-pqs6p"]
Jul 14 19:52:48.668850 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:48.668828    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pqs6p"
Jul 14 19:52:48.671630 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:48.671606    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 14 19:52:48.671950 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:48.671917    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-tstt4\""
Jul 14 19:52:48.675291 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:48.675229    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-pqs6p"]
Jul 14 19:52:48.736457 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:48.736423    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fccfa7ec-d7e0-4279-8824-94d22fd84ff2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pqs6p\" (UID: \"fccfa7ec-d7e0-4279-8824-94d22fd84ff2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pqs6p"
Jul 14 19:52:48.736632 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:48.736496    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lndlh\" (UniqueName: \"kubernetes.io/projected/fccfa7ec-d7e0-4279-8824-94d22fd84ff2-kube-api-access-lndlh\") pod \"llmisvc-controller-manager-777c6bd7d5-pqs6p\" (UID: \"fccfa7ec-d7e0-4279-8824-94d22fd84ff2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pqs6p"
Jul 14 19:52:48.837586 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:48.837530    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lndlh\" (UniqueName: \"kubernetes.io/projected/fccfa7ec-d7e0-4279-8824-94d22fd84ff2-kube-api-access-lndlh\") pod \"llmisvc-controller-manager-777c6bd7d5-pqs6p\" (UID: \"fccfa7ec-d7e0-4279-8824-94d22fd84ff2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pqs6p"
Jul 14 19:52:48.837773 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:48.837665    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fccfa7ec-d7e0-4279-8824-94d22fd84ff2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pqs6p\" (UID: \"fccfa7ec-d7e0-4279-8824-94d22fd84ff2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pqs6p"
Jul 14 19:52:48.840587 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:48.840557    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fccfa7ec-d7e0-4279-8824-94d22fd84ff2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pqs6p\" (UID: \"fccfa7ec-d7e0-4279-8824-94d22fd84ff2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pqs6p"
Jul 14 19:52:48.849262 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:48.849239    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lndlh\" (UniqueName: \"kubernetes.io/projected/fccfa7ec-d7e0-4279-8824-94d22fd84ff2-kube-api-access-lndlh\") pod \"llmisvc-controller-manager-777c6bd7d5-pqs6p\" (UID: \"fccfa7ec-d7e0-4279-8824-94d22fd84ff2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pqs6p"
Jul 14 19:52:48.983859 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:48.983765    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pqs6p"
Jul 14 19:52:49.431429 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:49.431395    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d8f87cd4c-769pv"]
Jul 14 19:52:49.434678 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:49.434654    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d8f87cd4c-769pv"
Jul 14 19:52:49.437685 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:49.437657    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 14 19:52:49.437792 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:49.437760    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 14 19:52:49.437792 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:49.437768    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 14 19:52:49.437951 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:49.437925    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-2jln2\""
Jul 14 19:52:49.439320 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:49.439306    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 14 19:52:49.442530 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:49.442507    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05036eb5-3926-42c7-916e-b631b97a599e-tmp\") pod \"kube-auth-proxy-5d8f87cd4c-769pv\" (UID: \"05036eb5-3926-42c7-916e-b631b97a599e\") " pod="openshift-ingress/kube-auth-proxy-5d8f87cd4c-769pv"
Jul 14 19:52:49.442640 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:49.442545    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnlr9\" (UniqueName: \"kubernetes.io/projected/05036eb5-3926-42c7-916e-b631b97a599e-kube-api-access-tnlr9\") pod \"kube-auth-proxy-5d8f87cd4c-769pv\" (UID: \"05036eb5-3926-42c7-916e-b631b97a599e\") " pod="openshift-ingress/kube-auth-proxy-5d8f87cd4c-769pv"
Jul 14 19:52:49.442640 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:49.442572    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05036eb5-3926-42c7-916e-b631b97a599e-tls-certs\") pod \"kube-auth-proxy-5d8f87cd4c-769pv\" (UID: \"05036eb5-3926-42c7-916e-b631b97a599e\") " pod="openshift-ingress/kube-auth-proxy-5d8f87cd4c-769pv"
Jul 14 19:52:49.450621 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:49.450590    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d8f87cd4c-769pv"]
Jul 14 19:52:49.543530 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:49.543500    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05036eb5-3926-42c7-916e-b631b97a599e-tmp\") pod \"kube-auth-proxy-5d8f87cd4c-769pv\" (UID: \"05036eb5-3926-42c7-916e-b631b97a599e\") " pod="openshift-ingress/kube-auth-proxy-5d8f87cd4c-769pv"
Jul 14 19:52:49.543659 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:49.543558    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnlr9\" (UniqueName: \"kubernetes.io/projected/05036eb5-3926-42c7-916e-b631b97a599e-kube-api-access-tnlr9\") pod \"kube-auth-proxy-5d8f87cd4c-769pv\" (UID: \"05036eb5-3926-42c7-916e-b631b97a599e\") " pod="openshift-ingress/kube-auth-proxy-5d8f87cd4c-769pv"
Jul 14 19:52:49.543659 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:49.543600    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05036eb5-3926-42c7-916e-b631b97a599e-tls-certs\") pod \"kube-auth-proxy-5d8f87cd4c-769pv\" (UID: \"05036eb5-3926-42c7-916e-b631b97a599e\") " pod="openshift-ingress/kube-auth-proxy-5d8f87cd4c-769pv"
Jul 14 19:52:49.543831 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:52:49.543811    2560 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 14 19:52:49.543899 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:52:49.543879    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05036eb5-3926-42c7-916e-b631b97a599e-tls-certs podName:05036eb5-3926-42c7-916e-b631b97a599e nodeName:}" failed. No retries permitted until 2026-07-14 19:52:50.043858913 +0000 UTC m=+322.644162608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/05036eb5-3926-42c7-916e-b631b97a599e-tls-certs") pod "kube-auth-proxy-5d8f87cd4c-769pv" (UID: "05036eb5-3926-42c7-916e-b631b97a599e") : secret "kube-auth-proxy-tls" not found
Jul 14 19:52:49.545797 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:49.545776    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05036eb5-3926-42c7-916e-b631b97a599e-tmp\") pod \"kube-auth-proxy-5d8f87cd4c-769pv\" (UID: \"05036eb5-3926-42c7-916e-b631b97a599e\") " pod="openshift-ingress/kube-auth-proxy-5d8f87cd4c-769pv"
Jul 14 19:52:49.553286 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:49.553259    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnlr9\" (UniqueName: \"kubernetes.io/projected/05036eb5-3926-42c7-916e-b631b97a599e-kube-api-access-tnlr9\") pod \"kube-auth-proxy-5d8f87cd4c-769pv\" (UID: \"05036eb5-3926-42c7-916e-b631b97a599e\") " pod="openshift-ingress/kube-auth-proxy-5d8f87cd4c-769pv"
Jul 14 19:52:49.576821 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:49.576794    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-pqs6p"]
Jul 14 19:52:49.579686 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:52:49.579658    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfccfa7ec_d7e0_4279_8824_94d22fd84ff2.slice/crio-6a116c74035c52460dc43a558c471c2415e8a353720ed88dccd7df54342270d0 WatchSource:0}: Error finding container 6a116c74035c52460dc43a558c471c2415e8a353720ed88dccd7df54342270d0: Status 404 returned error can't find the container with id 6a116c74035c52460dc43a558c471c2415e8a353720ed88dccd7df54342270d0
Jul 14 19:52:50.049593 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:50.049527    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05036eb5-3926-42c7-916e-b631b97a599e-tls-certs\") pod \"kube-auth-proxy-5d8f87cd4c-769pv\" (UID: \"05036eb5-3926-42c7-916e-b631b97a599e\") " pod="openshift-ingress/kube-auth-proxy-5d8f87cd4c-769pv"
Jul 14 19:52:50.052489 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:50.052464    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05036eb5-3926-42c7-916e-b631b97a599e-tls-certs\") pod \"kube-auth-proxy-5d8f87cd4c-769pv\" (UID: \"05036eb5-3926-42c7-916e-b631b97a599e\") " pod="openshift-ingress/kube-auth-proxy-5d8f87cd4c-769pv"
Jul 14 19:52:50.061312 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:50.061287    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d8f87cd4c-769pv"
Jul 14 19:52:50.202967 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:50.202926    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d8f87cd4c-769pv"]
Jul 14 19:52:50.226578 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:50.226494    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pqs6p" event={"ID":"fccfa7ec-d7e0-4279-8824-94d22fd84ff2","Type":"ContainerStarted","Data":"6a116c74035c52460dc43a558c471c2415e8a353720ed88dccd7df54342270d0"}
Jul 14 19:52:50.227937 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:50.227912    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-69z6h" event={"ID":"411bcb53-f540-49a8-8665-3b257034d8f4","Type":"ContainerStarted","Data":"370e485bf43f7ceca7b036ee21a8bc794a7dd41ff96a5d0eca0d9f346813fd77"}
Jul 14 19:52:50.228163 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:50.228145    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-69z6h"
Jul 14 19:52:50.231088 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:52:50.231066    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05036eb5_3926_42c7_916e_b631b97a599e.slice/crio-f067bbd5003bef8834f93c175ec31809b283cace004677392db528cbc672c77d WatchSource:0}: Error finding container f067bbd5003bef8834f93c175ec31809b283cace004677392db528cbc672c77d: Status 404 returned error can't find the container with id f067bbd5003bef8834f93c175ec31809b283cace004677392db528cbc672c77d
Jul 14 19:52:50.254001 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:50.253950    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-69z6h" podStartSLOduration=1.79231736 podStartE2EDuration="4.253934451s" totalImagesPullingTime="2.461617091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:52:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:52:50.252769011 +0000 UTC m=+322.853072719" watchObservedRunningTime="2026-07-14 19:52:50.253934451 +0000 UTC m=+322.854238154"
Jul 14 19:52:51.234958 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:51.234920    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d8f87cd4c-769pv" event={"ID":"05036eb5-3926-42c7-916e-b631b97a599e","Type":"ContainerStarted","Data":"f067bbd5003bef8834f93c175ec31809b283cace004677392db528cbc672c77d"}
Jul 14 19:52:52.242441 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:52.242404    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pqs6p" event={"ID":"fccfa7ec-d7e0-4279-8824-94d22fd84ff2","Type":"ContainerStarted","Data":"e75c8aba0ec479e825301441fb9b7cc54fe51e0792901b9980d174dcefe20820"}
Jul 14 19:52:52.242840 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:52.242519    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pqs6p"
Jul 14 19:52:52.260002 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:52.259947    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pqs6p" podStartSLOduration=1.7361948379999999 podStartE2EDuration="4.259930355s" totalImagesPullingTime="2.523735517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:52:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:52:52.257743834 +0000 UTC m=+324.858047538" watchObservedRunningTime="2026-07-14 19:52:52.259930355 +0000 UTC m=+324.860234075"
Jul 14 19:52:53.249434 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:53.249403    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d8f87cd4c-769pv" event={"ID":"05036eb5-3926-42c7-916e-b631b97a599e","Type":"ContainerStarted","Data":"cc6fe21800d3ba792ec14da4c6dc41d0c3b2904c69713e97ec9f126b2b339ed2"}
Jul 14 19:52:53.268370 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:53.268315    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d8f87cd4c-769pv" podStartSLOduration=1.349422091 podStartE2EDuration="4.268297921s" totalImagesPullingTime="2.91887583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:52:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:52:53.266558597 +0000 UTC m=+325.866862299" watchObservedRunningTime="2026-07-14 19:52:53.268297921 +0000 UTC m=+325.868601624"
Jul 14 19:52:54.199535 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:52:54.199505    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-jfck4"
Jul 14 19:53:01.237160 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:53:01.237129    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-69z6h"
Jul 14 19:53:23.251510 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:53:23.251480    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pqs6p"
Jul 14 19:54:26.774337 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:26.774301    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-8d7dq"]
Jul 14 19:54:26.776736 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:26.776717    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8d7dq"
Jul 14 19:54:26.779104 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:26.779082    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 14 19:54:26.779242 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:26.779087    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 14 19:54:26.780288 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:26.780269    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 14 19:54:26.780288 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:26.780280    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-h7c6z\""
Jul 14 19:54:26.780457 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:26.780319    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 14 19:54:26.785586 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:26.785566    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-8d7dq"]
Jul 14 19:54:26.897577 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:26.897546    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c0c2a600-5f9a-4d03-aabf-d28814e14cad-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-8d7dq\" (UID: \"c0c2a600-5f9a-4d03-aabf-d28814e14cad\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8d7dq"
Jul 14 19:54:26.897734 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:26.897587    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0c2a600-5f9a-4d03-aabf-d28814e14cad-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8d7dq\" (UID: \"c0c2a600-5f9a-4d03-aabf-d28814e14cad\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8d7dq"
Jul 14 19:54:26.897734 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:26.897605    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc9jj\" (UniqueName: \"kubernetes.io/projected/c0c2a600-5f9a-4d03-aabf-d28814e14cad-kube-api-access-qc9jj\") pod \"kuadrant-console-plugin-66b5774985-8d7dq\" (UID: \"c0c2a600-5f9a-4d03-aabf-d28814e14cad\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8d7dq"
Jul 14 19:54:26.999037 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:26.999006    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c0c2a600-5f9a-4d03-aabf-d28814e14cad-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-8d7dq\" (UID: \"c0c2a600-5f9a-4d03-aabf-d28814e14cad\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8d7dq"
Jul 14 19:54:26.999262 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:26.999053    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0c2a600-5f9a-4d03-aabf-d28814e14cad-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8d7dq\" (UID: \"c0c2a600-5f9a-4d03-aabf-d28814e14cad\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8d7dq"
Jul 14 19:54:26.999262 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:26.999080    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc9jj\" (UniqueName: \"kubernetes.io/projected/c0c2a600-5f9a-4d03-aabf-d28814e14cad-kube-api-access-qc9jj\") pod \"kuadrant-console-plugin-66b5774985-8d7dq\" (UID: \"c0c2a600-5f9a-4d03-aabf-d28814e14cad\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8d7dq"
Jul 14 19:54:26.999702 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:26.999675    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c0c2a600-5f9a-4d03-aabf-d28814e14cad-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-8d7dq\" (UID: \"c0c2a600-5f9a-4d03-aabf-d28814e14cad\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8d7dq"
Jul 14 19:54:27.001730 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:27.001702    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0c2a600-5f9a-4d03-aabf-d28814e14cad-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8d7dq\" (UID: \"c0c2a600-5f9a-4d03-aabf-d28814e14cad\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8d7dq"
Jul 14 19:54:27.007463 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:27.007438    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc9jj\" (UniqueName: \"kubernetes.io/projected/c0c2a600-5f9a-4d03-aabf-d28814e14cad-kube-api-access-qc9jj\") pod \"kuadrant-console-plugin-66b5774985-8d7dq\" (UID: \"c0c2a600-5f9a-4d03-aabf-d28814e14cad\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8d7dq"
Jul 14 19:54:27.087237 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:27.087195    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8d7dq"
Jul 14 19:54:27.218724 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:27.218702    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-8d7dq"]
Jul 14 19:54:27.221751 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:54:27.221716    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0c2a600_5f9a_4d03_aabf_d28814e14cad.slice/crio-95bd304b48d70f6e7ac134ba40dafd85e934a4cbe4b9d3935c22f9efd2023cc6 WatchSource:0}: Error finding container 95bd304b48d70f6e7ac134ba40dafd85e934a4cbe4b9d3935c22f9efd2023cc6: Status 404 returned error can't find the container with id 95bd304b48d70f6e7ac134ba40dafd85e934a4cbe4b9d3935c22f9efd2023cc6
Jul 14 19:54:27.645355 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:27.645322    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8d7dq" event={"ID":"c0c2a600-5f9a-4d03-aabf-d28814e14cad","Type":"ContainerStarted","Data":"95bd304b48d70f6e7ac134ba40dafd85e934a4cbe4b9d3935c22f9efd2023cc6"}
Jul 14 19:54:37.257047 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:37.257011    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tgrjq"]
Jul 14 19:54:37.291253 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:37.291220    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tgrjq"]
Jul 14 19:54:37.291443 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:37.291341    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tgrjq"
Jul 14 19:54:37.294132 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:37.294091    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-65fsx\""
Jul 14 19:54:37.392431 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:37.392388    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7816b379-f4d6-4090-a17d-20a66bfafd53-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-tgrjq\" (UID: \"7816b379-f4d6-4090-a17d-20a66bfafd53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tgrjq"
Jul 14 19:54:37.392612 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:37.392444    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpr4n\" (UniqueName: \"kubernetes.io/projected/7816b379-f4d6-4090-a17d-20a66bfafd53-kube-api-access-rpr4n\") pod \"kuadrant-operator-controller-manager-55467d9f7-tgrjq\" (UID: \"7816b379-f4d6-4090-a17d-20a66bfafd53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tgrjq"
Jul 14 19:54:37.493811 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:37.493760    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7816b379-f4d6-4090-a17d-20a66bfafd53-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-tgrjq\" (UID: \"7816b379-f4d6-4090-a17d-20a66bfafd53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tgrjq"
Jul 14 19:54:37.493811 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:37.493807    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpr4n\" (UniqueName: \"kubernetes.io/projected/7816b379-f4d6-4090-a17d-20a66bfafd53-kube-api-access-rpr4n\") pod \"kuadrant-operator-controller-manager-55467d9f7-tgrjq\" (UID: \"7816b379-f4d6-4090-a17d-20a66bfafd53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tgrjq"
Jul 14 19:54:37.494189 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:37.494152    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7816b379-f4d6-4090-a17d-20a66bfafd53-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-tgrjq\" (UID: \"7816b379-f4d6-4090-a17d-20a66bfafd53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tgrjq"
Jul 14 19:54:37.504764 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:37.504730    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpr4n\" (UniqueName: \"kubernetes.io/projected/7816b379-f4d6-4090-a17d-20a66bfafd53-kube-api-access-rpr4n\") pod \"kuadrant-operator-controller-manager-55467d9f7-tgrjq\" (UID: \"7816b379-f4d6-4090-a17d-20a66bfafd53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tgrjq"
Jul 14 19:54:37.604871 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:37.604827    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tgrjq"
Jul 14 19:54:48.366698 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:48.366675    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tgrjq"]
Jul 14 19:54:48.368938 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:54:48.368913    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7816b379_f4d6_4090_a17d_20a66bfafd53.slice/crio-5198bdc23bbfa403a28974646331aa5c724bb8dae3bc71827ec6bd96569a4f5e WatchSource:0}: Error finding container 5198bdc23bbfa403a28974646331aa5c724bb8dae3bc71827ec6bd96569a4f5e: Status 404 returned error can't find the container with id 5198bdc23bbfa403a28974646331aa5c724bb8dae3bc71827ec6bd96569a4f5e
Jul 14 19:54:48.751160 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:48.751124    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tgrjq" event={"ID":"7816b379-f4d6-4090-a17d-20a66bfafd53","Type":"ContainerStarted","Data":"5198bdc23bbfa403a28974646331aa5c724bb8dae3bc71827ec6bd96569a4f5e"}
Jul 14 19:54:48.752426 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:48.752405    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8d7dq" event={"ID":"c0c2a600-5f9a-4d03-aabf-d28814e14cad","Type":"ContainerStarted","Data":"a2006d6c6df307be40a793a6edc7c87c4e975fd9664c7681e2d4e5d9f94e60b1"}
Jul 14 19:54:48.769813 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:48.769263    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8d7dq" podStartSLOduration=1.6829981200000002 podStartE2EDuration="22.769242781s" totalImagesPullingTime="21.086244661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:54:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:54:48.768343861 +0000 UTC m=+441.368647576" watchObservedRunningTime="2026-07-14 19:54:48.769242781 +0000 UTC m=+441.369546487"
Jul 14 19:54:54.786137 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:54.786099    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tgrjq" event={"ID":"7816b379-f4d6-4090-a17d-20a66bfafd53","Type":"ContainerStarted","Data":"42ea4b924456972b6a02513b25f00a2fed4ed463e06a38ce00d58b628a0de849"}
Jul 14 19:54:54.786639 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:54.786218    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tgrjq"
Jul 14 19:54:54.810847 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:54:54.807604    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tgrjq" podStartSLOduration=12.496716303 podStartE2EDuration="17.807581521s" totalImagesPullingTime="5.310865218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:54:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:54:54.806084696 +0000 UTC m=+447.406388397" watchObservedRunningTime="2026-07-14 19:54:54.807581521 +0000 UTC m=+447.407885227"
Jul 14 19:55:05.792576 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:05.792547    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tgrjq"
Jul 14 19:55:09.476786 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:09.476754    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fbddf6585-p8j7l"]
Jul 14 19:55:18.688477 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:18.688446    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tgrjq"]
Jul 14 19:55:18.688934 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:18.688662    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tgrjq" podUID="7816b379-f4d6-4090-a17d-20a66bfafd53" containerName="manager" containerID="cri-o://42ea4b924456972b6a02513b25f00a2fed4ed463e06a38ce00d58b628a0de849" gracePeriod=10
Jul 14 19:55:18.897235 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:18.897199    2560 generic.go:356] "Generic (PLEG): container finished" podID="7816b379-f4d6-4090-a17d-20a66bfafd53" containerID="42ea4b924456972b6a02513b25f00a2fed4ed463e06a38ce00d58b628a0de849" exitCode=0
Jul 14 19:55:18.897380 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:18.897211    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tgrjq" event={"ID":"7816b379-f4d6-4090-a17d-20a66bfafd53","Type":"ContainerDied","Data":"42ea4b924456972b6a02513b25f00a2fed4ed463e06a38ce00d58b628a0de849"}
Jul 14 19:55:18.942117 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:18.942058    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tgrjq"
Jul 14 19:55:19.067400 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:19.067370    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpr4n\" (UniqueName: \"kubernetes.io/projected/7816b379-f4d6-4090-a17d-20a66bfafd53-kube-api-access-rpr4n\") pod \"7816b379-f4d6-4090-a17d-20a66bfafd53\" (UID: \"7816b379-f4d6-4090-a17d-20a66bfafd53\") "
Jul 14 19:55:19.067575 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:19.067500    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7816b379-f4d6-4090-a17d-20a66bfafd53-extensions-socket-volume\") pod \"7816b379-f4d6-4090-a17d-20a66bfafd53\" (UID: \"7816b379-f4d6-4090-a17d-20a66bfafd53\") "
Jul 14 19:55:19.067966 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:19.067936    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7816b379-f4d6-4090-a17d-20a66bfafd53-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7816b379-f4d6-4090-a17d-20a66bfafd53" (UID: "7816b379-f4d6-4090-a17d-20a66bfafd53"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:55:19.069681 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:19.069653    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7816b379-f4d6-4090-a17d-20a66bfafd53-kube-api-access-rpr4n" (OuterVolumeSpecName: "kube-api-access-rpr4n") pod "7816b379-f4d6-4090-a17d-20a66bfafd53" (UID: "7816b379-f4d6-4090-a17d-20a66bfafd53"). InnerVolumeSpecName "kube-api-access-rpr4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:55:19.168442 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:19.168397    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7816b379-f4d6-4090-a17d-20a66bfafd53-extensions-socket-volume\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:55:19.168442 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:19.168429    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpr4n\" (UniqueName: \"kubernetes.io/projected/7816b379-f4d6-4090-a17d-20a66bfafd53-kube-api-access-rpr4n\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:55:19.902466 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:19.902440    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tgrjq"
Jul 14 19:55:19.902466 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:19.902451    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tgrjq" event={"ID":"7816b379-f4d6-4090-a17d-20a66bfafd53","Type":"ContainerDied","Data":"5198bdc23bbfa403a28974646331aa5c724bb8dae3bc71827ec6bd96569a4f5e"}
Jul 14 19:55:19.902887 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:19.902505    2560 scope.go:117] "RemoveContainer" containerID="42ea4b924456972b6a02513b25f00a2fed4ed463e06a38ce00d58b628a0de849"
Jul 14 19:55:19.927357 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:19.927329    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tgrjq"]
Jul 14 19:55:19.933332 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:19.933311    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tgrjq"]
Jul 14 19:55:21.888636 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:21.888603    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7816b379-f4d6-4090-a17d-20a66bfafd53" path="/var/lib/kubelet/pods/7816b379-f4d6-4090-a17d-20a66bfafd53/volumes"
Jul 14 19:55:25.800237 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:25.800207    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-g7zv7"]
Jul 14 19:55:25.800725 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:25.800583    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7816b379-f4d6-4090-a17d-20a66bfafd53" containerName="manager"
Jul 14 19:55:25.800725 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:25.800599    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7816b379-f4d6-4090-a17d-20a66bfafd53" containerName="manager"
Jul 14 19:55:25.800725 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:25.800670    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7816b379-f4d6-4090-a17d-20a66bfafd53" containerName="manager"
Jul 14 19:55:25.865887 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:25.865855    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-g7zv7"]
Jul 14 19:55:25.866052 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:25.865947    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-g7zv7"
Jul 14 19:55:25.868429 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:25.868405    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-94dvd\""
Jul 14 19:55:25.922055 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:25.922012    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qtkf\" (UniqueName: \"kubernetes.io/projected/192e775b-c3b2-43bb-9e7e-1d182e4fa672-kube-api-access-4qtkf\") pod \"authorino-58cd955675-g7zv7\" (UID: \"192e775b-c3b2-43bb-9e7e-1d182e4fa672\") " pod="kuadrant-system/authorino-58cd955675-g7zv7"
Jul 14 19:55:26.001936 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:26.001900    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-hbs6v"]
Jul 14 19:55:26.005408 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:26.005387    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-hbs6v"
Jul 14 19:55:26.010627 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:26.010605    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-hbs6v"]
Jul 14 19:55:26.023029 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:26.023008    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcfr6\" (UniqueName: \"kubernetes.io/projected/baf21f23-6225-44c3-8e15-9fb32f4cd51d-kube-api-access-dcfr6\") pod \"authorino-75d556bd77-hbs6v\" (UID: \"baf21f23-6225-44c3-8e15-9fb32f4cd51d\") " pod="kuadrant-system/authorino-75d556bd77-hbs6v"
Jul 14 19:55:26.023136 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:26.023081    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qtkf\" (UniqueName: \"kubernetes.io/projected/192e775b-c3b2-43bb-9e7e-1d182e4fa672-kube-api-access-4qtkf\") pod \"authorino-58cd955675-g7zv7\" (UID: \"192e775b-c3b2-43bb-9e7e-1d182e4fa672\") " pod="kuadrant-system/authorino-58cd955675-g7zv7"
Jul 14 19:55:26.031441 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:26.031407    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qtkf\" (UniqueName: \"kubernetes.io/projected/192e775b-c3b2-43bb-9e7e-1d182e4fa672-kube-api-access-4qtkf\") pod \"authorino-58cd955675-g7zv7\" (UID: \"192e775b-c3b2-43bb-9e7e-1d182e4fa672\") " pod="kuadrant-system/authorino-58cd955675-g7zv7"
Jul 14 19:55:26.124509 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:26.124418    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcfr6\" (UniqueName: \"kubernetes.io/projected/baf21f23-6225-44c3-8e15-9fb32f4cd51d-kube-api-access-dcfr6\") pod \"authorino-75d556bd77-hbs6v\" (UID: \"baf21f23-6225-44c3-8e15-9fb32f4cd51d\") " pod="kuadrant-system/authorino-75d556bd77-hbs6v"
Jul 14 19:55:26.134035 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:26.134000    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcfr6\" (UniqueName: \"kubernetes.io/projected/baf21f23-6225-44c3-8e15-9fb32f4cd51d-kube-api-access-dcfr6\") pod \"authorino-75d556bd77-hbs6v\" (UID: \"baf21f23-6225-44c3-8e15-9fb32f4cd51d\") " pod="kuadrant-system/authorino-75d556bd77-hbs6v"
Jul 14 19:55:26.176693 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:26.176660    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-g7zv7"
Jul 14 19:55:26.303383 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:26.303278    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-g7zv7"]
Jul 14 19:55:26.306020 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:55:26.305994    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod192e775b_c3b2_43bb_9e7e_1d182e4fa672.slice/crio-bf17ed1f8f07f52419fed54b179af79ffd29c06fccd2b4fdf3122221822f22d2 WatchSource:0}: Error finding container bf17ed1f8f07f52419fed54b179af79ffd29c06fccd2b4fdf3122221822f22d2: Status 404 returned error can't find the container with id bf17ed1f8f07f52419fed54b179af79ffd29c06fccd2b4fdf3122221822f22d2
Jul 14 19:55:26.315772 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:26.315752    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-hbs6v"
Jul 14 19:55:26.442930 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:26.442901    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-hbs6v"]
Jul 14 19:55:26.445643 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:55:26.445623    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbaf21f23_6225_44c3_8e15_9fb32f4cd51d.slice/crio-c91ef2580bbf5a99e05e76207011e8c6bb20b809106473da287d48b77eeb14fa WatchSource:0}: Error finding container c91ef2580bbf5a99e05e76207011e8c6bb20b809106473da287d48b77eeb14fa: Status 404 returned error can't find the container with id c91ef2580bbf5a99e05e76207011e8c6bb20b809106473da287d48b77eeb14fa
Jul 14 19:55:26.934350 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:26.934312    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-g7zv7" event={"ID":"192e775b-c3b2-43bb-9e7e-1d182e4fa672","Type":"ContainerStarted","Data":"bf17ed1f8f07f52419fed54b179af79ffd29c06fccd2b4fdf3122221822f22d2"}
Jul 14 19:55:26.935279 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:26.935259    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-hbs6v" event={"ID":"baf21f23-6225-44c3-8e15-9fb32f4cd51d","Type":"ContainerStarted","Data":"c91ef2580bbf5a99e05e76207011e8c6bb20b809106473da287d48b77eeb14fa"}
Jul 14 19:55:30.959034 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:30.958997    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-g7zv7" event={"ID":"192e775b-c3b2-43bb-9e7e-1d182e4fa672","Type":"ContainerStarted","Data":"f65884706abeb71942aff3f74f4b0bc00f5596d7b7c4628832c2ddf8b4ad3e06"}
Jul 14 19:55:30.960501 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:30.960475    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-hbs6v" event={"ID":"baf21f23-6225-44c3-8e15-9fb32f4cd51d","Type":"ContainerStarted","Data":"13222b5b170591c70bac3beb5b6a2af35b51ecdfdd793cdf51f9c27367926f5f"}
Jul 14 19:55:30.975064 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:30.975016    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-g7zv7" podStartSLOduration=2.183044032 podStartE2EDuration="5.975000808s" totalImagesPullingTime="3.791956776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:55:30.974647504 +0000 UTC m=+483.574951208" watchObservedRunningTime="2026-07-14 19:55:30.975000808 +0000 UTC m=+483.575304510"
Jul 14 19:55:30.997785 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:30.994635    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-hbs6v" podStartSLOduration=2.331639179 podStartE2EDuration="5.994616183s" totalImagesPullingTime="3.662977004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:55:30.992085844 +0000 UTC m=+483.592389549" watchObservedRunningTime="2026-07-14 19:55:30.994616183 +0000 UTC m=+483.594919886"
Jul 14 19:55:31.019724 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:31.019692    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-g7zv7"]
Jul 14 19:55:32.969014 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:32.968967    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-g7zv7" podUID="192e775b-c3b2-43bb-9e7e-1d182e4fa672" containerName="authorino" containerID="cri-o://f65884706abeb71942aff3f74f4b0bc00f5596d7b7c4628832c2ddf8b4ad3e06" gracePeriod=30
Jul 14 19:55:33.218831 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:33.218804    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-g7zv7"
Jul 14 19:55:33.285295 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:33.285270    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qtkf\" (UniqueName: \"kubernetes.io/projected/192e775b-c3b2-43bb-9e7e-1d182e4fa672-kube-api-access-4qtkf\") pod \"192e775b-c3b2-43bb-9e7e-1d182e4fa672\" (UID: \"192e775b-c3b2-43bb-9e7e-1d182e4fa672\") "
Jul 14 19:55:33.287430 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:33.287407    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192e775b-c3b2-43bb-9e7e-1d182e4fa672-kube-api-access-4qtkf" (OuterVolumeSpecName: "kube-api-access-4qtkf") pod "192e775b-c3b2-43bb-9e7e-1d182e4fa672" (UID: "192e775b-c3b2-43bb-9e7e-1d182e4fa672"). InnerVolumeSpecName "kube-api-access-4qtkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:55:33.386042 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:33.385991    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qtkf\" (UniqueName: \"kubernetes.io/projected/192e775b-c3b2-43bb-9e7e-1d182e4fa672-kube-api-access-4qtkf\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:55:33.973553 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:33.973517    2560 generic.go:356] "Generic (PLEG): container finished" podID="192e775b-c3b2-43bb-9e7e-1d182e4fa672" containerID="f65884706abeb71942aff3f74f4b0bc00f5596d7b7c4628832c2ddf8b4ad3e06" exitCode=0
Jul 14 19:55:33.973938 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:33.973565    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-g7zv7"
Jul 14 19:55:33.973938 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:33.973610    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-g7zv7" event={"ID":"192e775b-c3b2-43bb-9e7e-1d182e4fa672","Type":"ContainerDied","Data":"f65884706abeb71942aff3f74f4b0bc00f5596d7b7c4628832c2ddf8b4ad3e06"}
Jul 14 19:55:33.973938 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:33.973657    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-g7zv7" event={"ID":"192e775b-c3b2-43bb-9e7e-1d182e4fa672","Type":"ContainerDied","Data":"bf17ed1f8f07f52419fed54b179af79ffd29c06fccd2b4fdf3122221822f22d2"}
Jul 14 19:55:33.973938 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:33.973675    2560 scope.go:117] "RemoveContainer" containerID="f65884706abeb71942aff3f74f4b0bc00f5596d7b7c4628832c2ddf8b4ad3e06"
Jul 14 19:55:33.981853 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:33.981833    2560 scope.go:117] "RemoveContainer" containerID="f65884706abeb71942aff3f74f4b0bc00f5596d7b7c4628832c2ddf8b4ad3e06"
Jul 14 19:55:33.982100 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:55:33.982082    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f65884706abeb71942aff3f74f4b0bc00f5596d7b7c4628832c2ddf8b4ad3e06\": container with ID starting with f65884706abeb71942aff3f74f4b0bc00f5596d7b7c4628832c2ddf8b4ad3e06 not found: ID does not exist" containerID="f65884706abeb71942aff3f74f4b0bc00f5596d7b7c4628832c2ddf8b4ad3e06"
Jul 14 19:55:33.982140 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:33.982111    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f65884706abeb71942aff3f74f4b0bc00f5596d7b7c4628832c2ddf8b4ad3e06"} err="failed to get container status \"f65884706abeb71942aff3f74f4b0bc00f5596d7b7c4628832c2ddf8b4ad3e06\": rpc error: code = NotFound desc = could not find container \"f65884706abeb71942aff3f74f4b0bc00f5596d7b7c4628832c2ddf8b4ad3e06\": container with ID starting with f65884706abeb71942aff3f74f4b0bc00f5596d7b7c4628832c2ddf8b4ad3e06 not found: ID does not exist"
Jul 14 19:55:33.991394 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:33.991367    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-g7zv7"]
Jul 14 19:55:33.993448 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:33.993428    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-g7zv7"]
Jul 14 19:55:34.497232 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.497187    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7fbddf6585-p8j7l" podUID="882a6bd1-e610-47c2-a5bf-7f5ed64be422" containerName="console" containerID="cri-o://08400d80f0200a429369e6d9141619c79f6ab02266d4fc0f9cf5e815707ab64f" gracePeriod=15
Jul 14 19:55:34.596719 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.596676    2560 patch_prober.go:28] interesting pod/console-7fbddf6585-p8j7l container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.16:8443/health\": dial tcp 10.133.0.16:8443: connect: connection refused" start-of-body=
Jul 14 19:55:34.596892 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.596728    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-7fbddf6585-p8j7l" podUID="882a6bd1-e610-47c2-a5bf-7f5ed64be422" containerName="console" probeResult="failure" output="Get \"https://10.133.0.16:8443/health\": dial tcp 10.133.0.16:8443: connect: connection refused"
Jul 14 19:55:34.766271 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.766237    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:55:34.798308 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.798271    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/882a6bd1-e610-47c2-a5bf-7f5ed64be422-console-config\") pod \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") "
Jul 14 19:55:34.798431 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.798319    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5d7w\" (UniqueName: \"kubernetes.io/projected/882a6bd1-e610-47c2-a5bf-7f5ed64be422-kube-api-access-x5d7w\") pod \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") "
Jul 14 19:55:34.798431 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.798356    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/882a6bd1-e610-47c2-a5bf-7f5ed64be422-service-ca\") pod \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") "
Jul 14 19:55:34.798431 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.798402    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/882a6bd1-e610-47c2-a5bf-7f5ed64be422-console-oauth-config\") pod \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") "
Jul 14 19:55:34.798431 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.798427    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/882a6bd1-e610-47c2-a5bf-7f5ed64be422-console-serving-cert\") pod \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") "
Jul 14 19:55:34.798642 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.798456    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/882a6bd1-e610-47c2-a5bf-7f5ed64be422-oauth-serving-cert\") pod \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") "
Jul 14 19:55:34.798642 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.798499    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/882a6bd1-e610-47c2-a5bf-7f5ed64be422-trusted-ca-bundle\") pod \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\" (UID: \"882a6bd1-e610-47c2-a5bf-7f5ed64be422\") "
Jul 14 19:55:34.798740 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.798712    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/882a6bd1-e610-47c2-a5bf-7f5ed64be422-console-config" (OuterVolumeSpecName: "console-config") pod "882a6bd1-e610-47c2-a5bf-7f5ed64be422" (UID: "882a6bd1-e610-47c2-a5bf-7f5ed64be422"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:55:34.798817 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.798785    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/882a6bd1-e610-47c2-a5bf-7f5ed64be422-service-ca" (OuterVolumeSpecName: "service-ca") pod "882a6bd1-e610-47c2-a5bf-7f5ed64be422" (UID: "882a6bd1-e610-47c2-a5bf-7f5ed64be422"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:55:34.799004 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.798975    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/882a6bd1-e610-47c2-a5bf-7f5ed64be422-console-config\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:55:34.799004 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.798992    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/882a6bd1-e610-47c2-a5bf-7f5ed64be422-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "882a6bd1-e610-47c2-a5bf-7f5ed64be422" (UID: "882a6bd1-e610-47c2-a5bf-7f5ed64be422"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:55:34.799165 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.799000    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/882a6bd1-e610-47c2-a5bf-7f5ed64be422-service-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:55:34.799165 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.799079    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/882a6bd1-e610-47c2-a5bf-7f5ed64be422-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "882a6bd1-e610-47c2-a5bf-7f5ed64be422" (UID: "882a6bd1-e610-47c2-a5bf-7f5ed64be422"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:55:34.800666 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.800644    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/882a6bd1-e610-47c2-a5bf-7f5ed64be422-kube-api-access-x5d7w" (OuterVolumeSpecName: "kube-api-access-x5d7w") pod "882a6bd1-e610-47c2-a5bf-7f5ed64be422" (UID: "882a6bd1-e610-47c2-a5bf-7f5ed64be422"). InnerVolumeSpecName "kube-api-access-x5d7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:55:34.801230 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.801202    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/882a6bd1-e610-47c2-a5bf-7f5ed64be422-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "882a6bd1-e610-47c2-a5bf-7f5ed64be422" (UID: "882a6bd1-e610-47c2-a5bf-7f5ed64be422"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:55:34.801230 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.801216    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/882a6bd1-e610-47c2-a5bf-7f5ed64be422-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "882a6bd1-e610-47c2-a5bf-7f5ed64be422" (UID: "882a6bd1-e610-47c2-a5bf-7f5ed64be422"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:55:34.899609 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.899578    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5d7w\" (UniqueName: \"kubernetes.io/projected/882a6bd1-e610-47c2-a5bf-7f5ed64be422-kube-api-access-x5d7w\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:55:34.899609 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.899603    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/882a6bd1-e610-47c2-a5bf-7f5ed64be422-console-oauth-config\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:55:34.899609 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.899612    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/882a6bd1-e610-47c2-a5bf-7f5ed64be422-console-serving-cert\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:55:34.899609 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.899620    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/882a6bd1-e610-47c2-a5bf-7f5ed64be422-oauth-serving-cert\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:55:34.899856 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.899628    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/882a6bd1-e610-47c2-a5bf-7f5ed64be422-trusted-ca-bundle\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:55:34.979755 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.979720    2560 generic.go:356] "Generic (PLEG): container finished" podID="882a6bd1-e610-47c2-a5bf-7f5ed64be422" containerID="08400d80f0200a429369e6d9141619c79f6ab02266d4fc0f9cf5e815707ab64f" exitCode=0
Jul 14 19:55:34.980138 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.979767    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbddf6585-p8j7l"
Jul 14 19:55:34.980138 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.979765    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbddf6585-p8j7l" event={"ID":"882a6bd1-e610-47c2-a5bf-7f5ed64be422","Type":"ContainerDied","Data":"08400d80f0200a429369e6d9141619c79f6ab02266d4fc0f9cf5e815707ab64f"}
Jul 14 19:55:34.980138 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.979884    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbddf6585-p8j7l" event={"ID":"882a6bd1-e610-47c2-a5bf-7f5ed64be422","Type":"ContainerDied","Data":"0ce18f04342068c116d3648d8af08ab39e831e9d7fd0511b4b66403273ead75e"}
Jul 14 19:55:34.980138 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.979913    2560 scope.go:117] "RemoveContainer" containerID="08400d80f0200a429369e6d9141619c79f6ab02266d4fc0f9cf5e815707ab64f"
Jul 14 19:55:34.988905 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.988884    2560 scope.go:117] "RemoveContainer" containerID="08400d80f0200a429369e6d9141619c79f6ab02266d4fc0f9cf5e815707ab64f"
Jul 14 19:55:34.989155 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:55:34.989138    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08400d80f0200a429369e6d9141619c79f6ab02266d4fc0f9cf5e815707ab64f\": container with ID starting with 08400d80f0200a429369e6d9141619c79f6ab02266d4fc0f9cf5e815707ab64f not found: ID does not exist" containerID="08400d80f0200a429369e6d9141619c79f6ab02266d4fc0f9cf5e815707ab64f"
Jul 14 19:55:34.989229 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:34.989163    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08400d80f0200a429369e6d9141619c79f6ab02266d4fc0f9cf5e815707ab64f"} err="failed to get container status \"08400d80f0200a429369e6d9141619c79f6ab02266d4fc0f9cf5e815707ab64f\": rpc error: code = NotFound desc = could not find container \"08400d80f0200a429369e6d9141619c79f6ab02266d4fc0f9cf5e815707ab64f\": container with ID starting with 08400d80f0200a429369e6d9141619c79f6ab02266d4fc0f9cf5e815707ab64f not found: ID does not exist"
Jul 14 19:55:35.003459 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:35.003400    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fbddf6585-p8j7l"]
Jul 14 19:55:35.007293 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:35.007273    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7fbddf6585-p8j7l"]
Jul 14 19:55:35.888996 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:35.888961    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="192e775b-c3b2-43bb-9e7e-1d182e4fa672" path="/var/lib/kubelet/pods/192e775b-c3b2-43bb-9e7e-1d182e4fa672/volumes"
Jul 14 19:55:35.889354 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:35.889341    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="882a6bd1-e610-47c2-a5bf-7f5ed64be422" path="/var/lib/kubelet/pods/882a6bd1-e610-47c2-a5bf-7f5ed64be422/volumes"
Jul 14 19:55:54.266874 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:54.266831    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-5j24x"]
Jul 14 19:55:54.267284 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:54.267151    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="882a6bd1-e610-47c2-a5bf-7f5ed64be422" containerName="console"
Jul 14 19:55:54.267284 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:54.267163    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="882a6bd1-e610-47c2-a5bf-7f5ed64be422" containerName="console"
Jul 14 19:55:54.267284 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:54.267187    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="192e775b-c3b2-43bb-9e7e-1d182e4fa672" containerName="authorino"
Jul 14 19:55:54.267284 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:54.267192    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="192e775b-c3b2-43bb-9e7e-1d182e4fa672" containerName="authorino"
Jul 14 19:55:54.267284 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:54.267252    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="882a6bd1-e610-47c2-a5bf-7f5ed64be422" containerName="console"
Jul 14 19:55:54.267284 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:54.267263    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="192e775b-c3b2-43bb-9e7e-1d182e4fa672" containerName="authorino"
Jul 14 19:55:54.270155 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:54.270137    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-5j24x"
Jul 14 19:55:54.277479 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:54.277459    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-5j24x"]
Jul 14 19:55:54.367267 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:54.367231    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72w44\" (UniqueName: \"kubernetes.io/projected/0dffe686-9895-419b-85fa-f6abef33f239-kube-api-access-72w44\") pod \"authorino-5d4b67c446-5j24x\" (UID: \"0dffe686-9895-419b-85fa-f6abef33f239\") " pod="kuadrant-system/authorino-5d4b67c446-5j24x"
Jul 14 19:55:54.467938 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:54.467909    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72w44\" (UniqueName: \"kubernetes.io/projected/0dffe686-9895-419b-85fa-f6abef33f239-kube-api-access-72w44\") pod \"authorino-5d4b67c446-5j24x\" (UID: \"0dffe686-9895-419b-85fa-f6abef33f239\") " pod="kuadrant-system/authorino-5d4b67c446-5j24x"
Jul 14 19:55:54.484105 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:54.484075    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72w44\" (UniqueName: \"kubernetes.io/projected/0dffe686-9895-419b-85fa-f6abef33f239-kube-api-access-72w44\") pod \"authorino-5d4b67c446-5j24x\" (UID: \"0dffe686-9895-419b-85fa-f6abef33f239\") " pod="kuadrant-system/authorino-5d4b67c446-5j24x"
Jul 14 19:55:54.550910 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:54.550872    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-5j24x"]
Jul 14 19:55:54.551129 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:54.551119    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-5j24x"
Jul 14 19:55:54.700662 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:54.700628    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-5j24x"]
Jul 14 19:55:54.703861 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:55:54.703831    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0dffe686_9895_419b_85fa_f6abef33f239.slice/crio-e9b2a50abff581ae694ecc73a50fdaef399cea07dc6dff4cd7d9f81a0a5e5852 WatchSource:0}: Error finding container e9b2a50abff581ae694ecc73a50fdaef399cea07dc6dff4cd7d9f81a0a5e5852: Status 404 returned error can't find the container with id e9b2a50abff581ae694ecc73a50fdaef399cea07dc6dff4cd7d9f81a0a5e5852
Jul 14 19:55:55.078167 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:55.078134    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-5j24x" event={"ID":"0dffe686-9895-419b-85fa-f6abef33f239","Type":"ContainerStarted","Data":"e9b2a50abff581ae694ecc73a50fdaef399cea07dc6dff4cd7d9f81a0a5e5852"}
Jul 14 19:55:56.083620 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:56.083584    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-5j24x" event={"ID":"0dffe686-9895-419b-85fa-f6abef33f239","Type":"ContainerStarted","Data":"3226f661004a1ae886eca0f927682c3c2ec3d0e7f1a5dd25d814fb10070a8b49"}
Jul 14 19:55:56.084076 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:56.083655    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-5j24x" podUID="0dffe686-9895-419b-85fa-f6abef33f239" containerName="authorino" containerID="cri-o://3226f661004a1ae886eca0f927682c3c2ec3d0e7f1a5dd25d814fb10070a8b49" gracePeriod=30
Jul 14 19:55:56.098954 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:56.098910    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-5j24x" podStartSLOduration=1.3915369819999999 podStartE2EDuration="2.098893591s" totalImagesPullingTime="707.356609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:55:56.097747396 +0000 UTC m=+508.698051099" watchObservedRunningTime="2026-07-14 19:55:56.098893591 +0000 UTC m=+508.699197294"
Jul 14 19:55:56.325561 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:56.325531    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-5j24x"
Jul 14 19:55:56.382809 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:56.382711    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72w44\" (UniqueName: \"kubernetes.io/projected/0dffe686-9895-419b-85fa-f6abef33f239-kube-api-access-72w44\") pod \"0dffe686-9895-419b-85fa-f6abef33f239\" (UID: \"0dffe686-9895-419b-85fa-f6abef33f239\") "
Jul 14 19:55:56.384901 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:56.384863    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dffe686-9895-419b-85fa-f6abef33f239-kube-api-access-72w44" (OuterVolumeSpecName: "kube-api-access-72w44") pod "0dffe686-9895-419b-85fa-f6abef33f239" (UID: "0dffe686-9895-419b-85fa-f6abef33f239"). InnerVolumeSpecName "kube-api-access-72w44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:55:56.483599 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:56.483564    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72w44\" (UniqueName: \"kubernetes.io/projected/0dffe686-9895-419b-85fa-f6abef33f239-kube-api-access-72w44\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:55:57.089646 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:57.089613    2560 generic.go:356] "Generic (PLEG): container finished" podID="0dffe686-9895-419b-85fa-f6abef33f239" containerID="3226f661004a1ae886eca0f927682c3c2ec3d0e7f1a5dd25d814fb10070a8b49" exitCode=0
Jul 14 19:55:57.090144 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:57.089655    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-5j24x"
Jul 14 19:55:57.090144 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:57.089676    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-5j24x" event={"ID":"0dffe686-9895-419b-85fa-f6abef33f239","Type":"ContainerDied","Data":"3226f661004a1ae886eca0f927682c3c2ec3d0e7f1a5dd25d814fb10070a8b49"}
Jul 14 19:55:57.090144 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:57.089715    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-5j24x" event={"ID":"0dffe686-9895-419b-85fa-f6abef33f239","Type":"ContainerDied","Data":"e9b2a50abff581ae694ecc73a50fdaef399cea07dc6dff4cd7d9f81a0a5e5852"}
Jul 14 19:55:57.090144 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:57.089737    2560 scope.go:117] "RemoveContainer" containerID="3226f661004a1ae886eca0f927682c3c2ec3d0e7f1a5dd25d814fb10070a8b49"
Jul 14 19:55:57.098135 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:57.098115    2560 scope.go:117] "RemoveContainer" containerID="3226f661004a1ae886eca0f927682c3c2ec3d0e7f1a5dd25d814fb10070a8b49"
Jul 14 19:55:57.098407 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:55:57.098388    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3226f661004a1ae886eca0f927682c3c2ec3d0e7f1a5dd25d814fb10070a8b49\": container with ID starting with 3226f661004a1ae886eca0f927682c3c2ec3d0e7f1a5dd25d814fb10070a8b49 not found: ID does not exist" containerID="3226f661004a1ae886eca0f927682c3c2ec3d0e7f1a5dd25d814fb10070a8b49"
Jul 14 19:55:57.098470 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:57.098419    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3226f661004a1ae886eca0f927682c3c2ec3d0e7f1a5dd25d814fb10070a8b49"} err="failed to get container status \"3226f661004a1ae886eca0f927682c3c2ec3d0e7f1a5dd25d814fb10070a8b49\": rpc error: code = NotFound desc = could not find container \"3226f661004a1ae886eca0f927682c3c2ec3d0e7f1a5dd25d814fb10070a8b49\": container with ID starting with 3226f661004a1ae886eca0f927682c3c2ec3d0e7f1a5dd25d814fb10070a8b49 not found: ID does not exist"
Jul 14 19:55:57.110029 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:57.110009    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-5j24x"]
Jul 14 19:55:57.114376 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:57.114355    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-5j24x"]
Jul 14 19:55:57.888522 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:57.888494    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dffe686-9895-419b-85fa-f6abef33f239" path="/var/lib/kubelet/pods/0dffe686-9895-419b-85fa-f6abef33f239/volumes"
Jul 14 19:55:58.925793 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:58.925760    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-hbs6v"]
Jul 14 19:55:58.926287 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:58.925990    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-hbs6v" podUID="baf21f23-6225-44c3-8e15-9fb32f4cd51d" containerName="authorino" containerID="cri-o://13222b5b170591c70bac3beb5b6a2af35b51ecdfdd793cdf51f9c27367926f5f" gracePeriod=30
Jul 14 19:55:59.100864 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:59.100836    2560 generic.go:356] "Generic (PLEG): container finished" podID="baf21f23-6225-44c3-8e15-9fb32f4cd51d" containerID="13222b5b170591c70bac3beb5b6a2af35b51ecdfdd793cdf51f9c27367926f5f" exitCode=0
Jul 14 19:55:59.100989 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:59.100906    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-hbs6v" event={"ID":"baf21f23-6225-44c3-8e15-9fb32f4cd51d","Type":"ContainerDied","Data":"13222b5b170591c70bac3beb5b6a2af35b51ecdfdd793cdf51f9c27367926f5f"}
Jul 14 19:55:59.172949 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:59.172926    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-hbs6v"
Jul 14 19:55:59.306084 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:59.306057    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcfr6\" (UniqueName: \"kubernetes.io/projected/baf21f23-6225-44c3-8e15-9fb32f4cd51d-kube-api-access-dcfr6\") pod \"baf21f23-6225-44c3-8e15-9fb32f4cd51d\" (UID: \"baf21f23-6225-44c3-8e15-9fb32f4cd51d\") "
Jul 14 19:55:59.308259 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:59.308232    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf21f23-6225-44c3-8e15-9fb32f4cd51d-kube-api-access-dcfr6" (OuterVolumeSpecName: "kube-api-access-dcfr6") pod "baf21f23-6225-44c3-8e15-9fb32f4cd51d" (UID: "baf21f23-6225-44c3-8e15-9fb32f4cd51d"). InnerVolumeSpecName "kube-api-access-dcfr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:55:59.407251 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:55:59.407224    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcfr6\" (UniqueName: \"kubernetes.io/projected/baf21f23-6225-44c3-8e15-9fb32f4cd51d-kube-api-access-dcfr6\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:56:00.106719 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:00.106685    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-hbs6v" event={"ID":"baf21f23-6225-44c3-8e15-9fb32f4cd51d","Type":"ContainerDied","Data":"c91ef2580bbf5a99e05e76207011e8c6bb20b809106473da287d48b77eeb14fa"}
Jul 14 19:56:00.106719 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:00.106703    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-hbs6v"
Jul 14 19:56:00.107274 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:00.106731    2560 scope.go:117] "RemoveContainer" containerID="13222b5b170591c70bac3beb5b6a2af35b51ecdfdd793cdf51f9c27367926f5f"
Jul 14 19:56:00.127410 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:00.127381    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-hbs6v"]
Jul 14 19:56:00.132343 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:00.132321    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-hbs6v"]
Jul 14 19:56:01.888553 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:01.888521    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baf21f23-6225-44c3-8e15-9fb32f4cd51d" path="/var/lib/kubelet/pods/baf21f23-6225-44c3-8e15-9fb32f4cd51d/volumes"
Jul 14 19:56:03.914848 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:03.914820    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8687d6b579-cpg8j"]
Jul 14 19:56:03.915270 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:03.915127    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf21f23-6225-44c3-8e15-9fb32f4cd51d" containerName="authorino"
Jul 14 19:56:03.915270 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:03.915138    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf21f23-6225-44c3-8e15-9fb32f4cd51d" containerName="authorino"
Jul 14 19:56:03.915270 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:03.915150    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dffe686-9895-419b-85fa-f6abef33f239" containerName="authorino"
Jul 14 19:56:03.915270 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:03.915155    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dffe686-9895-419b-85fa-f6abef33f239" containerName="authorino"
Jul 14 19:56:03.915270 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:03.915236    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dffe686-9895-419b-85fa-f6abef33f239" containerName="authorino"
Jul 14 19:56:03.915270 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:03.915247    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="baf21f23-6225-44c3-8e15-9fb32f4cd51d" containerName="authorino"
Jul 14 19:56:03.920404 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:03.920386    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8687d6b579-cpg8j"
Jul 14 19:56:03.922777 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:03.922752    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 14 19:56:03.922890 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:03.922781    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 14 19:56:03.922890 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:03.922825    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-g24q9\""
Jul 14 19:56:03.927255 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:03.927234    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8687d6b579-cpg8j"]
Jul 14 19:56:04.050465 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:04.050430    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5546daf5-a9b2-428a-9e6a-0f0e633ef47a-webhook-cert\") pod \"maas-controller-8687d6b579-cpg8j\" (UID: \"5546daf5-a9b2-428a-9e6a-0f0e633ef47a\") " pod="opendatahub/maas-controller-8687d6b579-cpg8j"
Jul 14 19:56:04.050645 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:04.050479    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxnhx\" (UniqueName: \"kubernetes.io/projected/5546daf5-a9b2-428a-9e6a-0f0e633ef47a-kube-api-access-fxnhx\") pod \"maas-controller-8687d6b579-cpg8j\" (UID: \"5546daf5-a9b2-428a-9e6a-0f0e633ef47a\") " pod="opendatahub/maas-controller-8687d6b579-cpg8j"
Jul 14 19:56:04.100780 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:04.100745    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b96486789-tnn57"]
Jul 14 19:56:04.105697 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:04.105679    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b96486789-tnn57"
Jul 14 19:56:04.117957 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:04.117379    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b96486789-tnn57"]
Jul 14 19:56:04.151257 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:04.151225    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5546daf5-a9b2-428a-9e6a-0f0e633ef47a-webhook-cert\") pod \"maas-controller-8687d6b579-cpg8j\" (UID: \"5546daf5-a9b2-428a-9e6a-0f0e633ef47a\") " pod="opendatahub/maas-controller-8687d6b579-cpg8j"
Jul 14 19:56:04.151401 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:04.151273    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxnhx\" (UniqueName: \"kubernetes.io/projected/5546daf5-a9b2-428a-9e6a-0f0e633ef47a-kube-api-access-fxnhx\") pod \"maas-controller-8687d6b579-cpg8j\" (UID: \"5546daf5-a9b2-428a-9e6a-0f0e633ef47a\") " pod="opendatahub/maas-controller-8687d6b579-cpg8j"
Jul 14 19:56:04.153781 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:04.153750    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5546daf5-a9b2-428a-9e6a-0f0e633ef47a-webhook-cert\") pod \"maas-controller-8687d6b579-cpg8j\" (UID: \"5546daf5-a9b2-428a-9e6a-0f0e633ef47a\") " pod="opendatahub/maas-controller-8687d6b579-cpg8j"
Jul 14 19:56:04.161722 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:04.161693    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxnhx\" (UniqueName: \"kubernetes.io/projected/5546daf5-a9b2-428a-9e6a-0f0e633ef47a-kube-api-access-fxnhx\") pod \"maas-controller-8687d6b579-cpg8j\" (UID: \"5546daf5-a9b2-428a-9e6a-0f0e633ef47a\") " pod="opendatahub/maas-controller-8687d6b579-cpg8j"
Jul 14 19:56:04.232264 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:04.232163    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8687d6b579-cpg8j"
Jul 14 19:56:04.252724 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:04.252689    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrtcm\" (UniqueName: \"kubernetes.io/projected/95db45a8-6be8-4f3d-aae1-898ac27cb2be-kube-api-access-hrtcm\") pod \"maas-controller-7b96486789-tnn57\" (UID: \"95db45a8-6be8-4f3d-aae1-898ac27cb2be\") " pod="opendatahub/maas-controller-7b96486789-tnn57"
Jul 14 19:56:04.252866 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:04.252757    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95db45a8-6be8-4f3d-aae1-898ac27cb2be-webhook-cert\") pod \"maas-controller-7b96486789-tnn57\" (UID: \"95db45a8-6be8-4f3d-aae1-898ac27cb2be\") " pod="opendatahub/maas-controller-7b96486789-tnn57"
Jul 14 19:56:04.353720 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:04.353688    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95db45a8-6be8-4f3d-aae1-898ac27cb2be-webhook-cert\") pod \"maas-controller-7b96486789-tnn57\" (UID: \"95db45a8-6be8-4f3d-aae1-898ac27cb2be\") " pod="opendatahub/maas-controller-7b96486789-tnn57"
Jul 14 19:56:04.353905 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:04.353776    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrtcm\" (UniqueName: \"kubernetes.io/projected/95db45a8-6be8-4f3d-aae1-898ac27cb2be-kube-api-access-hrtcm\") pod \"maas-controller-7b96486789-tnn57\" (UID: \"95db45a8-6be8-4f3d-aae1-898ac27cb2be\") " pod="opendatahub/maas-controller-7b96486789-tnn57"
Jul 14 19:56:04.357613 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:04.357542    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95db45a8-6be8-4f3d-aae1-898ac27cb2be-webhook-cert\") pod \"maas-controller-7b96486789-tnn57\" (UID: \"95db45a8-6be8-4f3d-aae1-898ac27cb2be\") " pod="opendatahub/maas-controller-7b96486789-tnn57"
Jul 14 19:56:04.363113 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:04.363088    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrtcm\" (UniqueName: \"kubernetes.io/projected/95db45a8-6be8-4f3d-aae1-898ac27cb2be-kube-api-access-hrtcm\") pod \"maas-controller-7b96486789-tnn57\" (UID: \"95db45a8-6be8-4f3d-aae1-898ac27cb2be\") " pod="opendatahub/maas-controller-7b96486789-tnn57"
Jul 14 19:56:04.387037 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:04.387009    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8687d6b579-cpg8j"]
Jul 14 19:56:04.390268 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:56:04.390240    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5546daf5_a9b2_428a_9e6a_0f0e633ef47a.slice/crio-10b0f71faf10fc050c2f0b438aa8ed72e7fc91676d7af4997543df0fb5a7a38f WatchSource:0}: Error finding container 10b0f71faf10fc050c2f0b438aa8ed72e7fc91676d7af4997543df0fb5a7a38f: Status 404 returned error can't find the container with id 10b0f71faf10fc050c2f0b438aa8ed72e7fc91676d7af4997543df0fb5a7a38f
Jul 14 19:56:04.422350 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:04.422325    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b96486789-tnn57"
Jul 14 19:56:04.585459 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:04.585428    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b96486789-tnn57"]
Jul 14 19:56:04.589036 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:56:04.589008    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95db45a8_6be8_4f3d_aae1_898ac27cb2be.slice/crio-7599df59e40428a03e97978884458b8beb1d822d1924098aa2bb3321e3c3b133 WatchSource:0}: Error finding container 7599df59e40428a03e97978884458b8beb1d822d1924098aa2bb3321e3c3b133: Status 404 returned error can't find the container with id 7599df59e40428a03e97978884458b8beb1d822d1924098aa2bb3321e3c3b133
Jul 14 19:56:05.131544 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:05.131504    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b96486789-tnn57" event={"ID":"95db45a8-6be8-4f3d-aae1-898ac27cb2be","Type":"ContainerStarted","Data":"7599df59e40428a03e97978884458b8beb1d822d1924098aa2bb3321e3c3b133"}
Jul 14 19:56:05.132752 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:05.132730    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8687d6b579-cpg8j" event={"ID":"5546daf5-a9b2-428a-9e6a-0f0e633ef47a","Type":"ContainerStarted","Data":"10b0f71faf10fc050c2f0b438aa8ed72e7fc91676d7af4997543df0fb5a7a38f"}
Jul 14 19:56:07.146723 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:07.146688    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b96486789-tnn57" event={"ID":"95db45a8-6be8-4f3d-aae1-898ac27cb2be","Type":"ContainerStarted","Data":"9604bbba207ba97f62775233cf4430cc951efeaa46c62a97b1db31737c2879d0"}
Jul 14 19:56:07.147190 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:07.146786    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b96486789-tnn57"
Jul 14 19:56:07.148357 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:07.148333    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8687d6b579-cpg8j" event={"ID":"5546daf5-a9b2-428a-9e6a-0f0e633ef47a","Type":"ContainerStarted","Data":"64849a40c6815d4a152022a61ce477b96a2c70c0d2a67bf647b8c155526c63b5"}
Jul 14 19:56:07.148454 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:07.148410    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8687d6b579-cpg8j"
Jul 14 19:56:07.168912 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:07.168871    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7b96486789-tnn57" podStartSLOduration=0.814236352 podStartE2EDuration="3.168858311s" totalImagesPullingTime="2.354621959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:56:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:56:07.167641461 +0000 UTC m=+519.767945164" watchObservedRunningTime="2026-07-14 19:56:07.168858311 +0000 UTC m=+519.769162014"
Jul 14 19:56:07.187984 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:07.187937    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-8687d6b579-cpg8j" podStartSLOduration=1.624350396 podStartE2EDuration="4.187922893s" totalImagesPullingTime="2.563572497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:56:07.187152796 +0000 UTC m=+519.787456518" watchObservedRunningTime="2026-07-14 19:56:07.187922893 +0000 UTC m=+519.788226595"
Jul 14 19:56:18.154566 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:18.154493    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8687d6b579-cpg8j"
Jul 14 19:56:18.154959 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:18.154579    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b96486789-tnn57"
Jul 14 19:56:18.215977 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:18.215883    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8687d6b579-cpg8j"]
Jul 14 19:56:18.216144 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:18.216089    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-8687d6b579-cpg8j" podUID="5546daf5-a9b2-428a-9e6a-0f0e633ef47a" containerName="manager" containerID="cri-o://64849a40c6815d4a152022a61ce477b96a2c70c0d2a67bf647b8c155526c63b5" gracePeriod=30
Jul 14 19:56:18.460733 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:18.460707    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8687d6b579-cpg8j"
Jul 14 19:56:18.475042 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:18.474981    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5546daf5-a9b2-428a-9e6a-0f0e633ef47a-webhook-cert\") pod \"5546daf5-a9b2-428a-9e6a-0f0e633ef47a\" (UID: \"5546daf5-a9b2-428a-9e6a-0f0e633ef47a\") "
Jul 14 19:56:18.475042 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:18.475034    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxnhx\" (UniqueName: \"kubernetes.io/projected/5546daf5-a9b2-428a-9e6a-0f0e633ef47a-kube-api-access-fxnhx\") pod \"5546daf5-a9b2-428a-9e6a-0f0e633ef47a\" (UID: \"5546daf5-a9b2-428a-9e6a-0f0e633ef47a\") "
Jul 14 19:56:18.477580 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:18.477552    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5546daf5-a9b2-428a-9e6a-0f0e633ef47a-kube-api-access-fxnhx" (OuterVolumeSpecName: "kube-api-access-fxnhx") pod "5546daf5-a9b2-428a-9e6a-0f0e633ef47a" (UID: "5546daf5-a9b2-428a-9e6a-0f0e633ef47a"). InnerVolumeSpecName "kube-api-access-fxnhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:56:18.477580 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:18.477564    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5546daf5-a9b2-428a-9e6a-0f0e633ef47a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "5546daf5-a9b2-428a-9e6a-0f0e633ef47a" (UID: "5546daf5-a9b2-428a-9e6a-0f0e633ef47a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:56:18.575804 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:18.575759    2560 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5546daf5-a9b2-428a-9e6a-0f0e633ef47a-webhook-cert\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:56:18.575804 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:18.575800    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxnhx\" (UniqueName: \"kubernetes.io/projected/5546daf5-a9b2-428a-9e6a-0f0e633ef47a-kube-api-access-fxnhx\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 19:56:19.207443 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:19.207407    2560 generic.go:356] "Generic (PLEG): container finished" podID="5546daf5-a9b2-428a-9e6a-0f0e633ef47a" containerID="64849a40c6815d4a152022a61ce477b96a2c70c0d2a67bf647b8c155526c63b5" exitCode=0
Jul 14 19:56:19.207855 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:19.207448    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8687d6b579-cpg8j"
Jul 14 19:56:19.207855 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:19.207460    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8687d6b579-cpg8j" event={"ID":"5546daf5-a9b2-428a-9e6a-0f0e633ef47a","Type":"ContainerDied","Data":"64849a40c6815d4a152022a61ce477b96a2c70c0d2a67bf647b8c155526c63b5"}
Jul 14 19:56:19.207855 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:19.207503    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8687d6b579-cpg8j" event={"ID":"5546daf5-a9b2-428a-9e6a-0f0e633ef47a","Type":"ContainerDied","Data":"10b0f71faf10fc050c2f0b438aa8ed72e7fc91676d7af4997543df0fb5a7a38f"}
Jul 14 19:56:19.207855 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:19.207524    2560 scope.go:117] "RemoveContainer" containerID="64849a40c6815d4a152022a61ce477b96a2c70c0d2a67bf647b8c155526c63b5"
Jul 14 19:56:19.217013 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:19.216888    2560 scope.go:117] "RemoveContainer" containerID="64849a40c6815d4a152022a61ce477b96a2c70c0d2a67bf647b8c155526c63b5"
Jul 14 19:56:19.217236 ip-10-0-133-168 kubenswrapper[2560]: E0714 19:56:19.217201    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64849a40c6815d4a152022a61ce477b96a2c70c0d2a67bf647b8c155526c63b5\": container with ID starting with 64849a40c6815d4a152022a61ce477b96a2c70c0d2a67bf647b8c155526c63b5 not found: ID does not exist" containerID="64849a40c6815d4a152022a61ce477b96a2c70c0d2a67bf647b8c155526c63b5"
Jul 14 19:56:19.217280 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:19.217229    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64849a40c6815d4a152022a61ce477b96a2c70c0d2a67bf647b8c155526c63b5"} err="failed to get container status \"64849a40c6815d4a152022a61ce477b96a2c70c0d2a67bf647b8c155526c63b5\": rpc error: code = NotFound desc = could not find container \"64849a40c6815d4a152022a61ce477b96a2c70c0d2a67bf647b8c155526c63b5\": container with ID starting with 64849a40c6815d4a152022a61ce477b96a2c70c0d2a67bf647b8c155526c63b5 not found: ID does not exist"
Jul 14 19:56:19.229420 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:19.229392    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8687d6b579-cpg8j"]
Jul 14 19:56:19.233231 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:19.233209    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8687d6b579-cpg8j"]
Jul 14 19:56:19.888988 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:19.888957    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5546daf5-a9b2-428a-9e6a-0f0e633ef47a" path="/var/lib/kubelet/pods/5546daf5-a9b2-428a-9e6a-0f0e633ef47a/volumes"
Jul 14 19:56:39.581968 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:39.581926    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7c45c66c84-h28pw"]
Jul 14 19:56:39.582441 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:39.582319    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5546daf5-a9b2-428a-9e6a-0f0e633ef47a" containerName="manager"
Jul 14 19:56:39.582441 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:39.582332    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5546daf5-a9b2-428a-9e6a-0f0e633ef47a" containerName="manager"
Jul 14 19:56:39.582441 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:39.582389    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5546daf5-a9b2-428a-9e6a-0f0e633ef47a" containerName="manager"
Jul 14 19:56:39.586741 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:39.586721    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c45c66c84-h28pw"
Jul 14 19:56:39.589648 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:39.589626    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 19:56:39.589768 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:39.589748    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 19:56:39.589865 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:39.589850    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 14 19:56:39.597320 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:39.597300    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c45c66c84-h28pw"]
Jul 14 19:56:39.653228 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:39.653193    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed092f93-6175-400c-b4a2-6ece6b791610-openshift-service-ca\") pod \"maas-api-7c45c66c84-h28pw\" (UID: \"ed092f93-6175-400c-b4a2-6ece6b791610\") " pod="odh-ai-gateway-infra/maas-api-7c45c66c84-h28pw"
Jul 14 19:56:39.653391 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:39.653256    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed092f93-6175-400c-b4a2-6ece6b791610-maas-api-tls\") pod \"maas-api-7c45c66c84-h28pw\" (UID: \"ed092f93-6175-400c-b4a2-6ece6b791610\") " pod="odh-ai-gateway-infra/maas-api-7c45c66c84-h28pw"
Jul 14 19:56:39.653391 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:39.653282    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prwvz\" (UniqueName: \"kubernetes.io/projected/ed092f93-6175-400c-b4a2-6ece6b791610-kube-api-access-prwvz\") pod \"maas-api-7c45c66c84-h28pw\" (UID: \"ed092f93-6175-400c-b4a2-6ece6b791610\") " pod="odh-ai-gateway-infra/maas-api-7c45c66c84-h28pw"
Jul 14 19:56:39.754692 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:39.754655    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed092f93-6175-400c-b4a2-6ece6b791610-maas-api-tls\") pod \"maas-api-7c45c66c84-h28pw\" (UID: \"ed092f93-6175-400c-b4a2-6ece6b791610\") " pod="odh-ai-gateway-infra/maas-api-7c45c66c84-h28pw"
Jul 14 19:56:39.754692 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:39.754696    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prwvz\" (UniqueName: \"kubernetes.io/projected/ed092f93-6175-400c-b4a2-6ece6b791610-kube-api-access-prwvz\") pod \"maas-api-7c45c66c84-h28pw\" (UID: \"ed092f93-6175-400c-b4a2-6ece6b791610\") " pod="odh-ai-gateway-infra/maas-api-7c45c66c84-h28pw"
Jul 14 19:56:39.754889 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:39.754743    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed092f93-6175-400c-b4a2-6ece6b791610-openshift-service-ca\") pod \"maas-api-7c45c66c84-h28pw\" (UID: \"ed092f93-6175-400c-b4a2-6ece6b791610\") " pod="odh-ai-gateway-infra/maas-api-7c45c66c84-h28pw"
Jul 14 19:56:39.755352 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:39.755328    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed092f93-6175-400c-b4a2-6ece6b791610-openshift-service-ca\") pod \"maas-api-7c45c66c84-h28pw\" (UID: \"ed092f93-6175-400c-b4a2-6ece6b791610\") " pod="odh-ai-gateway-infra/maas-api-7c45c66c84-h28pw"
Jul 14 19:56:39.757305 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:39.757277    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed092f93-6175-400c-b4a2-6ece6b791610-maas-api-tls\") pod \"maas-api-7c45c66c84-h28pw\" (UID: \"ed092f93-6175-400c-b4a2-6ece6b791610\") " pod="odh-ai-gateway-infra/maas-api-7c45c66c84-h28pw"
Jul 14 19:56:39.764434 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:39.764409    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prwvz\" (UniqueName: \"kubernetes.io/projected/ed092f93-6175-400c-b4a2-6ece6b791610-kube-api-access-prwvz\") pod \"maas-api-7c45c66c84-h28pw\" (UID: \"ed092f93-6175-400c-b4a2-6ece6b791610\") " pod="odh-ai-gateway-infra/maas-api-7c45c66c84-h28pw"
Jul 14 19:56:39.897360 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:39.897288    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c45c66c84-h28pw"
Jul 14 19:56:40.034197 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:40.033985    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c45c66c84-h28pw"]
Jul 14 19:56:40.036711 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:56:40.036672    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded092f93_6175_400c_b4a2_6ece6b791610.slice/crio-7ff25778405f6aefdfc44cf1dba249517b1a239a9a6cf7025a5f12c3617fca22 WatchSource:0}: Error finding container 7ff25778405f6aefdfc44cf1dba249517b1a239a9a6cf7025a5f12c3617fca22: Status 404 returned error can't find the container with id 7ff25778405f6aefdfc44cf1dba249517b1a239a9a6cf7025a5f12c3617fca22
Jul 14 19:56:40.175039 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:40.174960    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-57755"]
Jul 14 19:56:40.179722 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:40.179693    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-57755"
Jul 14 19:56:40.182079 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:40.182060    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 14 19:56:40.188557 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:40.188533    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-57755"]
Jul 14 19:56:40.224275 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:40.224246    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-rwkrz"]
Jul 14 19:56:40.227529 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:40.227511    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-rwkrz"
Jul 14 19:56:40.229938 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:40.229923    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-sp5d4\""
Jul 14 19:56:40.239262 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:40.239239    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-rwkrz"]
Jul 14 19:56:40.259156 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:40.259120    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aafb96c1-4030-49f7-8927-3ef681049bf0-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-57755\" (UID: \"aafb96c1-4030-49f7-8927-3ef681049bf0\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-57755"
Jul 14 19:56:40.259156 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:40.259157    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhh2p\" (UniqueName: \"kubernetes.io/projected/aafb96c1-4030-49f7-8927-3ef681049bf0-kube-api-access-mhh2p\") pod \"payload-pre-processing-74584bb9f8-57755\" (UID: \"aafb96c1-4030-49f7-8927-3ef681049bf0\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-57755"
Jul 14 19:56:40.304132 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:40.304098    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c45c66c84-h28pw" event={"ID":"ed092f93-6175-400c-b4a2-6ece6b791610","Type":"ContainerStarted","Data":"7ff25778405f6aefdfc44cf1dba249517b1a239a9a6cf7025a5f12c3617fca22"}
Jul 14 19:56:40.359744 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:40.359696    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5799959e-dffa-4835-94fb-945fbeae2967-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-rwkrz\" (UID: \"5799959e-dffa-4835-94fb-945fbeae2967\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-rwkrz"
Jul 14 19:56:40.359935 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:40.359847    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aafb96c1-4030-49f7-8927-3ef681049bf0-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-57755\" (UID: \"aafb96c1-4030-49f7-8927-3ef681049bf0\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-57755"
Jul 14 19:56:40.359935 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:40.359891    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5t9k\" (UniqueName: \"kubernetes.io/projected/5799959e-dffa-4835-94fb-945fbeae2967-kube-api-access-k5t9k\") pod \"payload-processing-f9b8d9b9b-rwkrz\" (UID: \"5799959e-dffa-4835-94fb-945fbeae2967\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-rwkrz"
Jul 14 19:56:40.359935 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:40.359921    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhh2p\" (UniqueName: \"kubernetes.io/projected/aafb96c1-4030-49f7-8927-3ef681049bf0-kube-api-access-mhh2p\") pod \"payload-pre-processing-74584bb9f8-57755\" (UID: \"aafb96c1-4030-49f7-8927-3ef681049bf0\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-57755"
Jul 14 19:56:40.360805 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:40.360781    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aafb96c1-4030-49f7-8927-3ef681049bf0-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-57755\" (UID: \"aafb96c1-4030-49f7-8927-3ef681049bf0\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-57755"
Jul 14 19:56:40.368509 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:40.368488    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhh2p\" (UniqueName: \"kubernetes.io/projected/aafb96c1-4030-49f7-8927-3ef681049bf0-kube-api-access-mhh2p\") pod \"payload-pre-processing-74584bb9f8-57755\" (UID: \"aafb96c1-4030-49f7-8927-3ef681049bf0\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-57755"
Jul 14 19:56:40.460796 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:40.460712    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5t9k\" (UniqueName: \"kubernetes.io/projected/5799959e-dffa-4835-94fb-945fbeae2967-kube-api-access-k5t9k\") pod \"payload-processing-f9b8d9b9b-rwkrz\" (UID: \"5799959e-dffa-4835-94fb-945fbeae2967\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-rwkrz"
Jul 14 19:56:40.460796 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:40.460774    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5799959e-dffa-4835-94fb-945fbeae2967-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-rwkrz\" (UID: \"5799959e-dffa-4835-94fb-945fbeae2967\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-rwkrz"
Jul 14 19:56:40.461441 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:40.461422    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5799959e-dffa-4835-94fb-945fbeae2967-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-rwkrz\" (UID: \"5799959e-dffa-4835-94fb-945fbeae2967\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-rwkrz"
Jul 14 19:56:40.470207 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:40.470158    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5t9k\" (UniqueName: \"kubernetes.io/projected/5799959e-dffa-4835-94fb-945fbeae2967-kube-api-access-k5t9k\") pod \"payload-processing-f9b8d9b9b-rwkrz\" (UID: \"5799959e-dffa-4835-94fb-945fbeae2967\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-rwkrz"
Jul 14 19:56:40.490973 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:40.490951    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-57755"
Jul 14 19:56:40.537266 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:40.537201    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-rwkrz"
Jul 14 19:56:40.630023 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:40.629914    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-57755"]
Jul 14 19:56:40.633044 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:56:40.632996    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaafb96c1_4030_49f7_8927_3ef681049bf0.slice/crio-be7a6c200ff120b9cccd6c2449f43600fedc45bc77e3afd2b91faf85b277a034 WatchSource:0}: Error finding container be7a6c200ff120b9cccd6c2449f43600fedc45bc77e3afd2b91faf85b277a034: Status 404 returned error can't find the container with id be7a6c200ff120b9cccd6c2449f43600fedc45bc77e3afd2b91faf85b277a034
Jul 14 19:56:40.682124 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:40.682094    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-rwkrz"]
Jul 14 19:56:40.684408 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:56:40.684378    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5799959e_dffa_4835_94fb_945fbeae2967.slice/crio-b4dd066af9af22f3d2630cca2f59dbf05c8953ce665d1b7134d972ee391535a9 WatchSource:0}: Error finding container b4dd066af9af22f3d2630cca2f59dbf05c8953ce665d1b7134d972ee391535a9: Status 404 returned error can't find the container with id b4dd066af9af22f3d2630cca2f59dbf05c8953ce665d1b7134d972ee391535a9
Jul 14 19:56:41.311695 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:41.311649    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-rwkrz" event={"ID":"5799959e-dffa-4835-94fb-945fbeae2967","Type":"ContainerStarted","Data":"b4dd066af9af22f3d2630cca2f59dbf05c8953ce665d1b7134d972ee391535a9"}
Jul 14 19:56:41.313115 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:41.313086    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-57755" event={"ID":"aafb96c1-4030-49f7-8927-3ef681049bf0","Type":"ContainerStarted","Data":"be7a6c200ff120b9cccd6c2449f43600fedc45bc77e3afd2b91faf85b277a034"}
Jul 14 19:56:42.319420 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:42.319333    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-57755" event={"ID":"aafb96c1-4030-49f7-8927-3ef681049bf0","Type":"ContainerStarted","Data":"1b11ca57680e87a3240d20c65d2b885bb9ada3991c510e81b85380391552f1be"}
Jul 14 19:56:42.319797 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:42.319456    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-57755"
Jul 14 19:56:42.321102 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:42.321077    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-rwkrz" event={"ID":"5799959e-dffa-4835-94fb-945fbeae2967","Type":"ContainerStarted","Data":"b3f5e954f1e694f03beae714d741120afef12fffeb8e5e69957632c0e2077931"}
Jul 14 19:56:42.321251 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:42.321192    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-rwkrz"
Jul 14 19:56:42.322532 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:42.322510    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c45c66c84-h28pw" event={"ID":"ed092f93-6175-400c-b4a2-6ece6b791610","Type":"ContainerStarted","Data":"a44e23c96182981017b841f28a3dbc5220fcd6fb06ab48108bd44997804d8d09"}
Jul 14 19:56:42.322611 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:42.322584    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7c45c66c84-h28pw"
Jul 14 19:56:42.335608 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:42.335567    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-57755" podStartSLOduration=0.901055431 podStartE2EDuration="2.33555384s" totalImagesPullingTime="1.434498409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:56:42.334110712 +0000 UTC m=+554.934414415" watchObservedRunningTime="2026-07-14 19:56:42.33555384 +0000 UTC m=+554.935857542"
Jul 14 19:56:42.349760 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:42.349719    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-f9b8d9b9b-rwkrz" podStartSLOduration=0.963863948 podStartE2EDuration="2.349707277s" totalImagesPullingTime="1.385843329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:56:42.348529714 +0000 UTC m=+554.948833417" watchObservedRunningTime="2026-07-14 19:56:42.349707277 +0000 UTC m=+554.950010979"
Jul 14 19:56:42.366469 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:42.366431    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7c45c66c84-h28pw" podStartSLOduration=1.335225839 podStartE2EDuration="3.366416291s" totalImagesPullingTime="2.031190452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:56:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:56:42.364652008 +0000 UTC m=+554.964955711" watchObservedRunningTime="2026-07-14 19:56:42.366416291 +0000 UTC m=+554.966719994"
Jul 14 19:56:48.333119 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:48.333092    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7c45c66c84-h28pw"
Jul 14 19:56:53.329983 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:53.329954    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-rwkrz"
Jul 14 19:56:53.330469 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:56:53.330103    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-57755"
Jul 14 19:57:27.818915 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:27.818889    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tzr6_4a36af4c-9eac-465f-9a0e-d65a382530f7/ovn-acl-logging/0.log"
Jul 14 19:57:27.819752 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:27.819731    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tzr6_4a36af4c-9eac-465f-9a0e-d65a382530f7/ovn-acl-logging/0.log"
Jul 14 19:57:32.350901 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:32.350856    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb"]
Jul 14 19:57:32.354816 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:32.354798    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb"
Jul 14 19:57:32.357384 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:32.357364    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 14 19:57:32.358223 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:32.358207    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 14 19:57:32.358291 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:32.358267    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 14 19:57:32.358394 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:32.358379    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xg6qp\""
Jul 14 19:57:32.362624 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:32.362601    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb"]
Jul 14 19:57:32.422426 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:32.422393    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ecff220-68c4-4ac9-a7a5-727e9bccfabf-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k6mhb\" (UID: \"7ecff220-68c4-4ac9-a7a5-727e9bccfabf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb"
Jul 14 19:57:32.422426 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:32.422435    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ecff220-68c4-4ac9-a7a5-727e9bccfabf-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k6mhb\" (UID: \"7ecff220-68c4-4ac9-a7a5-727e9bccfabf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb"
Jul 14 19:57:32.422643 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:32.422536    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ecff220-68c4-4ac9-a7a5-727e9bccfabf-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k6mhb\" (UID: \"7ecff220-68c4-4ac9-a7a5-727e9bccfabf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb"
Jul 14 19:57:32.422643 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:32.422561    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmpnh\" (UniqueName: \"kubernetes.io/projected/7ecff220-68c4-4ac9-a7a5-727e9bccfabf-kube-api-access-wmpnh\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k6mhb\" (UID: \"7ecff220-68c4-4ac9-a7a5-727e9bccfabf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb"
Jul 14 19:57:32.422643 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:32.422598    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ecff220-68c4-4ac9-a7a5-727e9bccfabf-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k6mhb\" (UID: \"7ecff220-68c4-4ac9-a7a5-727e9bccfabf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb"
Jul 14 19:57:32.422643 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:32.422618    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ecff220-68c4-4ac9-a7a5-727e9bccfabf-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k6mhb\" (UID: \"7ecff220-68c4-4ac9-a7a5-727e9bccfabf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb"
Jul 14 19:57:32.523446 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:32.523414    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmpnh\" (UniqueName: \"kubernetes.io/projected/7ecff220-68c4-4ac9-a7a5-727e9bccfabf-kube-api-access-wmpnh\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k6mhb\" (UID: \"7ecff220-68c4-4ac9-a7a5-727e9bccfabf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb"
Jul 14 19:57:32.523607 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:32.523456    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ecff220-68c4-4ac9-a7a5-727e9bccfabf-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k6mhb\" (UID: \"7ecff220-68c4-4ac9-a7a5-727e9bccfabf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb"
Jul 14 19:57:32.523607 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:32.523482    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ecff220-68c4-4ac9-a7a5-727e9bccfabf-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k6mhb\" (UID: \"7ecff220-68c4-4ac9-a7a5-727e9bccfabf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb"
Jul 14 19:57:32.523607 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:32.523521    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ecff220-68c4-4ac9-a7a5-727e9bccfabf-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k6mhb\" (UID: \"7ecff220-68c4-4ac9-a7a5-727e9bccfabf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb"
Jul 14 19:57:32.523607 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:32.523546    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ecff220-68c4-4ac9-a7a5-727e9bccfabf-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k6mhb\" (UID: \"7ecff220-68c4-4ac9-a7a5-727e9bccfabf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb"
Jul 14 19:57:32.523773 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:32.523616    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ecff220-68c4-4ac9-a7a5-727e9bccfabf-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k6mhb\" (UID: \"7ecff220-68c4-4ac9-a7a5-727e9bccfabf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb"
Jul 14 19:57:32.523945 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:32.523923    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ecff220-68c4-4ac9-a7a5-727e9bccfabf-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k6mhb\" (UID: \"7ecff220-68c4-4ac9-a7a5-727e9bccfabf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb"
Jul 14 19:57:32.524001 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:32.523979    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ecff220-68c4-4ac9-a7a5-727e9bccfabf-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k6mhb\" (UID: \"7ecff220-68c4-4ac9-a7a5-727e9bccfabf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb"
Jul 14 19:57:32.524059 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:32.524002    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ecff220-68c4-4ac9-a7a5-727e9bccfabf-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k6mhb\" (UID: \"7ecff220-68c4-4ac9-a7a5-727e9bccfabf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb"
Jul 14 19:57:32.525887 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:32.525858    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ecff220-68c4-4ac9-a7a5-727e9bccfabf-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k6mhb\" (UID: \"7ecff220-68c4-4ac9-a7a5-727e9bccfabf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb"
Jul 14 19:57:32.526228 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:32.526166    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ecff220-68c4-4ac9-a7a5-727e9bccfabf-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k6mhb\" (UID: \"7ecff220-68c4-4ac9-a7a5-727e9bccfabf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb"
Jul 14 19:57:32.531609 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:32.531589    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmpnh\" (UniqueName: \"kubernetes.io/projected/7ecff220-68c4-4ac9-a7a5-727e9bccfabf-kube-api-access-wmpnh\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k6mhb\" (UID: \"7ecff220-68c4-4ac9-a7a5-727e9bccfabf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb"
Jul 14 19:57:32.666159 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:32.666061    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb"
Jul 14 19:57:32.813394 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:32.813361    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb"]
Jul 14 19:57:32.816743 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:57:32.816713    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecff220_68c4_4ac9_a7a5_727e9bccfabf.slice/crio-3bf4ae2a4c83b0a40bbb0083cd5831353702f3d79a9a8d6d3b35d6f221638d52 WatchSource:0}: Error finding container 3bf4ae2a4c83b0a40bbb0083cd5831353702f3d79a9a8d6d3b35d6f221638d52: Status 404 returned error can't find the container with id 3bf4ae2a4c83b0a40bbb0083cd5831353702f3d79a9a8d6d3b35d6f221638d52
Jul 14 19:57:33.558872 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:33.558819    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb" event={"ID":"7ecff220-68c4-4ac9-a7a5-727e9bccfabf","Type":"ContainerStarted","Data":"3bf4ae2a4c83b0a40bbb0083cd5831353702f3d79a9a8d6d3b35d6f221638d52"}
Jul 14 19:57:34.565181 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:34.565146    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb" event={"ID":"7ecff220-68c4-4ac9-a7a5-727e9bccfabf","Type":"ContainerStarted","Data":"485e8ba7ec4bba09180ddda77684b5d1d96c7a5f8b1d219168086e8272a94fed"}
Jul 14 19:57:34.582969 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:34.582922    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb" podStartSLOduration=1.524716668 podStartE2EDuration="2.582906322s" totalImagesPullingTime="1.058189654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:57:34.581725301 +0000 UTC m=+607.182029041" watchObservedRunningTime="2026-07-14 19:57:34.582906322 +0000 UTC m=+607.183210208"
Jul 14 19:57:42.666612 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:42.666575    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb"
Jul 14 19:57:42.669297 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:42.666823    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb"
Jul 14 19:57:42.679317 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:42.679296    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb"
Jul 14 19:57:43.621852 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:43.621823    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k6mhb"
Jul 14 19:57:50.033748 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:50.033671    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v"]
Jul 14 19:57:50.037734 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:50.037714    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v"
Jul 14 19:57:50.040093 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:50.040067    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 14 19:57:50.046104 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:50.046082    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v"]
Jul 14 19:57:50.185941 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:50.185905    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0832b36-4701-4c38-85de-25654036ce78-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v\" (UID: \"a0832b36-4701-4c38-85de-25654036ce78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v"
Jul 14 19:57:50.186103 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:50.185962    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0832b36-4701-4c38-85de-25654036ce78-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v\" (UID: \"a0832b36-4701-4c38-85de-25654036ce78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v"
Jul 14 19:57:50.186103 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:50.186011    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0832b36-4701-4c38-85de-25654036ce78-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v\" (UID: \"a0832b36-4701-4c38-85de-25654036ce78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v"
Jul 14 19:57:50.186103 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:50.186049    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tgjj\" (UniqueName: \"kubernetes.io/projected/a0832b36-4701-4c38-85de-25654036ce78-kube-api-access-7tgjj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v\" (UID: \"a0832b36-4701-4c38-85de-25654036ce78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v"
Jul 14 19:57:50.186103 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:50.186083    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0832b36-4701-4c38-85de-25654036ce78-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v\" (UID: \"a0832b36-4701-4c38-85de-25654036ce78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v"
Jul 14 19:57:50.186384 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:50.186111    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0832b36-4701-4c38-85de-25654036ce78-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v\" (UID: \"a0832b36-4701-4c38-85de-25654036ce78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v"
Jul 14 19:57:50.286776 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:50.286685    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tgjj\" (UniqueName: \"kubernetes.io/projected/a0832b36-4701-4c38-85de-25654036ce78-kube-api-access-7tgjj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v\" (UID: \"a0832b36-4701-4c38-85de-25654036ce78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v"
Jul 14 19:57:50.286776 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:50.286731    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0832b36-4701-4c38-85de-25654036ce78-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v\" (UID: \"a0832b36-4701-4c38-85de-25654036ce78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v"
Jul 14 19:57:50.286776 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:50.286759    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0832b36-4701-4c38-85de-25654036ce78-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v\" (UID: \"a0832b36-4701-4c38-85de-25654036ce78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v"
Jul 14 19:57:50.287054 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:50.286790    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0832b36-4701-4c38-85de-25654036ce78-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v\" (UID: \"a0832b36-4701-4c38-85de-25654036ce78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v"
Jul 14 19:57:50.287054 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:50.286834    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0832b36-4701-4c38-85de-25654036ce78-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v\" (UID: \"a0832b36-4701-4c38-85de-25654036ce78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v"
Jul 14 19:57:50.287054 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:50.286866    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0832b36-4701-4c38-85de-25654036ce78-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v\" (UID: \"a0832b36-4701-4c38-85de-25654036ce78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v"
Jul 14 19:57:50.287259 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:50.287200    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0832b36-4701-4c38-85de-25654036ce78-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v\" (UID: \"a0832b36-4701-4c38-85de-25654036ce78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v"
Jul 14 19:57:50.287259 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:50.287245    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0832b36-4701-4c38-85de-25654036ce78-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v\" (UID: \"a0832b36-4701-4c38-85de-25654036ce78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v"
Jul 14 19:57:50.287360 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:50.287268    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0832b36-4701-4c38-85de-25654036ce78-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v\" (UID: \"a0832b36-4701-4c38-85de-25654036ce78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v"
Jul 14 19:57:50.289480 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:50.289455    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0832b36-4701-4c38-85de-25654036ce78-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v\" (UID: \"a0832b36-4701-4c38-85de-25654036ce78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v"
Jul 14 19:57:50.289480 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:50.289470    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0832b36-4701-4c38-85de-25654036ce78-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v\" (UID: \"a0832b36-4701-4c38-85de-25654036ce78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v"
Jul 14 19:57:50.295840 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:50.295821    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tgjj\" (UniqueName: \"kubernetes.io/projected/a0832b36-4701-4c38-85de-25654036ce78-kube-api-access-7tgjj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v\" (UID: \"a0832b36-4701-4c38-85de-25654036ce78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v"
Jul 14 19:57:50.350065 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:50.350024    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v"
Jul 14 19:57:50.488888 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:50.488841    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v"]
Jul 14 19:57:50.491922 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:57:50.491895    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0832b36_4701_4c38_85de_25654036ce78.slice/crio-0efc7a9c37dd857ca6ea1771be4b979174533a99cbf657fb4f635d7574dda75a WatchSource:0}: Error finding container 0efc7a9c37dd857ca6ea1771be4b979174533a99cbf657fb4f635d7574dda75a: Status 404 returned error can't find the container with id 0efc7a9c37dd857ca6ea1771be4b979174533a99cbf657fb4f635d7574dda75a
Jul 14 19:57:50.493683 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:50.493666    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 19:57:50.643836 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:50.643758    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v" event={"ID":"a0832b36-4701-4c38-85de-25654036ce78","Type":"ContainerStarted","Data":"0efc7a9c37dd857ca6ea1771be4b979174533a99cbf657fb4f635d7574dda75a"}
Jul 14 19:57:51.649465 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:51.649430    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v" event={"ID":"a0832b36-4701-4c38-85de-25654036ce78","Type":"ContainerStarted","Data":"fbe3564dd60e12abe3101c867f37624dd4b737631ed78a8d8e3c40cf55d024fc"}
Jul 14 19:57:51.668343 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:57:51.668294    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v" podStartSLOduration=1.46875737 podStartE2EDuration="1.668278222s" totalImagesPullingTime="199.520852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:57:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:57:51.666545118 +0000 UTC m=+624.266848821" watchObservedRunningTime="2026-07-14 19:57:51.668278222 +0000 UTC m=+624.268581904"
Jul 14 19:58:00.350758 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:00.350723    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v"
Jul 14 19:58:00.351312 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:00.350779    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v"
Jul 14 19:58:00.363275 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:00.363248    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v"
Jul 14 19:58:00.698903 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:00.698824    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v"
Jul 14 19:58:17.666445 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:17.666414    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g"]
Jul 14 19:58:17.672687 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:17.672665    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g"
Jul 14 19:58:17.675232 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:17.675214    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 14 19:58:17.679594 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:17.679570    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g"]
Jul 14 19:58:17.835444 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:17.835414    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a351dd-0cdd-475f-9909-9a6613594aa6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g\" (UID: \"99a351dd-0cdd-475f-9909-9a6613594aa6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g"
Jul 14 19:58:17.835444 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:17.835447    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jglhs\" (UniqueName: \"kubernetes.io/projected/99a351dd-0cdd-475f-9909-9a6613594aa6-kube-api-access-jglhs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g\" (UID: \"99a351dd-0cdd-475f-9909-9a6613594aa6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g"
Jul 14 19:58:17.835668 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:17.835478    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a351dd-0cdd-475f-9909-9a6613594aa6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g\" (UID: \"99a351dd-0cdd-475f-9909-9a6613594aa6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g"
Jul 14 19:58:17.835668 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:17.835544    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a351dd-0cdd-475f-9909-9a6613594aa6-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g\" (UID: \"99a351dd-0cdd-475f-9909-9a6613594aa6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g"
Jul 14 19:58:17.835668 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:17.835620    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a351dd-0cdd-475f-9909-9a6613594aa6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g\" (UID: \"99a351dd-0cdd-475f-9909-9a6613594aa6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g"
Jul 14 19:58:17.835668 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:17.835642    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a351dd-0cdd-475f-9909-9a6613594aa6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g\" (UID: \"99a351dd-0cdd-475f-9909-9a6613594aa6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g"
Jul 14 19:58:17.936560 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:17.936534    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a351dd-0cdd-475f-9909-9a6613594aa6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g\" (UID: \"99a351dd-0cdd-475f-9909-9a6613594aa6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g"
Jul 14 19:58:17.936666 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:17.936567    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jglhs\" (UniqueName: \"kubernetes.io/projected/99a351dd-0cdd-475f-9909-9a6613594aa6-kube-api-access-jglhs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g\" (UID: \"99a351dd-0cdd-475f-9909-9a6613594aa6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g"
Jul 14 19:58:17.936666 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:17.936596    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a351dd-0cdd-475f-9909-9a6613594aa6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g\" (UID: \"99a351dd-0cdd-475f-9909-9a6613594aa6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g"
Jul 14 19:58:17.936666 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:17.936618    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a351dd-0cdd-475f-9909-9a6613594aa6-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g\" (UID: \"99a351dd-0cdd-475f-9909-9a6613594aa6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g"
Jul 14 19:58:17.936666 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:17.936657    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a351dd-0cdd-475f-9909-9a6613594aa6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g\" (UID: \"99a351dd-0cdd-475f-9909-9a6613594aa6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g"
Jul 14 19:58:17.936825 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:17.936672    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a351dd-0cdd-475f-9909-9a6613594aa6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g\" (UID: \"99a351dd-0cdd-475f-9909-9a6613594aa6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g"
Jul 14 19:58:17.937013 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:17.936990    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a351dd-0cdd-475f-9909-9a6613594aa6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g\" (UID: \"99a351dd-0cdd-475f-9909-9a6613594aa6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g"
Jul 14 19:58:17.937088 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:17.937002    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a351dd-0cdd-475f-9909-9a6613594aa6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g\" (UID: \"99a351dd-0cdd-475f-9909-9a6613594aa6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g"
Jul 14 19:58:17.937088 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:17.937075    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a351dd-0cdd-475f-9909-9a6613594aa6-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g\" (UID: \"99a351dd-0cdd-475f-9909-9a6613594aa6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g"
Jul 14 19:58:17.938962 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:17.938944    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a351dd-0cdd-475f-9909-9a6613594aa6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g\" (UID: \"99a351dd-0cdd-475f-9909-9a6613594aa6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g"
Jul 14 19:58:17.939311 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:17.939292    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a351dd-0cdd-475f-9909-9a6613594aa6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g\" (UID: \"99a351dd-0cdd-475f-9909-9a6613594aa6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g"
Jul 14 19:58:17.945222 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:17.945195    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jglhs\" (UniqueName: \"kubernetes.io/projected/99a351dd-0cdd-475f-9909-9a6613594aa6-kube-api-access-jglhs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g\" (UID: \"99a351dd-0cdd-475f-9909-9a6613594aa6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g"
Jul 14 19:58:17.983647 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:17.983622    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g"
Jul 14 19:58:18.121905 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:18.121876    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g"]
Jul 14 19:58:18.123506 ip-10-0-133-168 kubenswrapper[2560]: W0714 19:58:18.123479    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99a351dd_0cdd_475f_9909_9a6613594aa6.slice/crio-fe99a6d1a51b7929de7b8df80f42b49f615091e6bf8433e6404aaf63b4702f07 WatchSource:0}: Error finding container fe99a6d1a51b7929de7b8df80f42b49f615091e6bf8433e6404aaf63b4702f07: Status 404 returned error can't find the container with id fe99a6d1a51b7929de7b8df80f42b49f615091e6bf8433e6404aaf63b4702f07
Jul 14 19:58:18.768635 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:18.768540    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g" event={"ID":"99a351dd-0cdd-475f-9909-9a6613594aa6","Type":"ContainerStarted","Data":"c69c5b4fab438ad040d873f48c2be439fbbe6f0caf97e94f1c0a1c55a66cba0c"}
Jul 14 19:58:18.768635 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:18.768589    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g" event={"ID":"99a351dd-0cdd-475f-9909-9a6613594aa6","Type":"ContainerStarted","Data":"fe99a6d1a51b7929de7b8df80f42b49f615091e6bf8433e6404aaf63b4702f07"}
Jul 14 19:58:18.787057 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:18.787007    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g" podStartSLOduration=1.617148373 podStartE2EDuration="1.786986648s" totalImagesPullingTime="169.838275ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:58:18.785414392 +0000 UTC m=+651.385718098" watchObservedRunningTime="2026-07-14 19:58:18.786986648 +0000 UTC m=+651.387290352"
Jul 14 19:58:27.984363 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:27.984327    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g"
Jul 14 19:58:27.984363 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:27.984371    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g"
Jul 14 19:58:27.999775 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:27.999740    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g"
Jul 14 19:58:28.829081 ip-10-0-133-168 kubenswrapper[2560]: I0714 19:58:28.829056    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g"
Jul 14 20:00:00.147553 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:00.147516    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734320-59n9c"]
Jul 14 20:00:00.151093 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:00.151076    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734320-59n9c"
Jul 14 20:00:00.153802 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:00.153772    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-56w6s\""
Jul 14 20:00:00.158744 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:00.158710    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734320-59n9c"]
Jul 14 20:00:00.162304 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:00.162284    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734320-59n9c"
Jul 14 20:00:00.306316 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:00.306293    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734320-59n9c"]
Jul 14 20:00:00.308678 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:00:00.308654    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ef60bd_641a_4e12_b9a5_ee3b819a19fe.slice/crio-c656fffb6dff47f9acbc016892ee07a19a6b38de9c4a7ea717f1471e3ce922b5 WatchSource:0}: Error finding container c656fffb6dff47f9acbc016892ee07a19a6b38de9c4a7ea717f1471e3ce922b5: Status 404 returned error can't find the container with id c656fffb6dff47f9acbc016892ee07a19a6b38de9c4a7ea717f1471e3ce922b5
Jul 14 20:00:01.246157 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:01.246122    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734320-59n9c" event={"ID":"67ef60bd-641a-4e12-b9a5-ee3b819a19fe","Type":"ContainerStarted","Data":"c656fffb6dff47f9acbc016892ee07a19a6b38de9c4a7ea717f1471e3ce922b5"}
Jul 14 20:00:06.270729 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:06.270696    2560 generic.go:356] "Generic (PLEG): container finished" podID="67ef60bd-641a-4e12-b9a5-ee3b819a19fe" containerID="0ac0db1aed390b6cc5e26559f3cf7c875853f3a2bbcfc416145bb14123b535d7" exitCode=0
Jul 14 20:00:06.271121 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:06.270777    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734320-59n9c" event={"ID":"67ef60bd-641a-4e12-b9a5-ee3b819a19fe","Type":"ContainerDied","Data":"0ac0db1aed390b6cc5e26559f3cf7c875853f3a2bbcfc416145bb14123b535d7"}
Jul 14 20:00:07.410001 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:07.409976    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734320-59n9c"
Jul 14 20:00:08.281874 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:08.281847    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734320-59n9c"
Jul 14 20:00:08.282028 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:08.281879    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734320-59n9c" event={"ID":"67ef60bd-641a-4e12-b9a5-ee3b819a19fe","Type":"ContainerDied","Data":"c656fffb6dff47f9acbc016892ee07a19a6b38de9c4a7ea717f1471e3ce922b5"}
Jul 14 20:00:08.282028 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:08.281920    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c656fffb6dff47f9acbc016892ee07a19a6b38de9c4a7ea717f1471e3ce922b5"
Jul 14 20:00:31.271724 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:31.271688    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b96486789-tnn57"]
Jul 14 20:00:31.272341 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:31.271945    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7b96486789-tnn57" podUID="95db45a8-6be8-4f3d-aae1-898ac27cb2be" containerName="manager" containerID="cri-o://9604bbba207ba97f62775233cf4430cc951efeaa46c62a97b1db31737c2879d0" gracePeriod=30
Jul 14 20:00:31.392733 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:31.392706    2560 generic.go:356] "Generic (PLEG): container finished" podID="95db45a8-6be8-4f3d-aae1-898ac27cb2be" containerID="9604bbba207ba97f62775233cf4430cc951efeaa46c62a97b1db31737c2879d0" exitCode=0
Jul 14 20:00:31.392842 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:31.392781    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b96486789-tnn57" event={"ID":"95db45a8-6be8-4f3d-aae1-898ac27cb2be","Type":"ContainerDied","Data":"9604bbba207ba97f62775233cf4430cc951efeaa46c62a97b1db31737c2879d0"}
Jul 14 20:00:31.513958 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:31.513930    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b96486789-tnn57"
Jul 14 20:00:31.662857 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:31.662825    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95db45a8-6be8-4f3d-aae1-898ac27cb2be-webhook-cert\") pod \"95db45a8-6be8-4f3d-aae1-898ac27cb2be\" (UID: \"95db45a8-6be8-4f3d-aae1-898ac27cb2be\") "
Jul 14 20:00:31.663029 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:31.662879    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrtcm\" (UniqueName: \"kubernetes.io/projected/95db45a8-6be8-4f3d-aae1-898ac27cb2be-kube-api-access-hrtcm\") pod \"95db45a8-6be8-4f3d-aae1-898ac27cb2be\" (UID: \"95db45a8-6be8-4f3d-aae1-898ac27cb2be\") "
Jul 14 20:00:31.665001 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:31.664968    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95db45a8-6be8-4f3d-aae1-898ac27cb2be-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "95db45a8-6be8-4f3d-aae1-898ac27cb2be" (UID: "95db45a8-6be8-4f3d-aae1-898ac27cb2be"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 20:00:31.665114 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:31.665016    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95db45a8-6be8-4f3d-aae1-898ac27cb2be-kube-api-access-hrtcm" (OuterVolumeSpecName: "kube-api-access-hrtcm") pod "95db45a8-6be8-4f3d-aae1-898ac27cb2be" (UID: "95db45a8-6be8-4f3d-aae1-898ac27cb2be"). InnerVolumeSpecName "kube-api-access-hrtcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 20:00:31.764463 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:31.764429    2560 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95db45a8-6be8-4f3d-aae1-898ac27cb2be-webhook-cert\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:00:31.764463 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:31.764457    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrtcm\" (UniqueName: \"kubernetes.io/projected/95db45a8-6be8-4f3d-aae1-898ac27cb2be-kube-api-access-hrtcm\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:00:32.398120 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:32.398084    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b96486789-tnn57" event={"ID":"95db45a8-6be8-4f3d-aae1-898ac27cb2be","Type":"ContainerDied","Data":"7599df59e40428a03e97978884458b8beb1d822d1924098aa2bb3321e3c3b133"}
Jul 14 20:00:32.398120 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:32.398116    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b96486789-tnn57"
Jul 14 20:00:32.398668 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:32.398155    2560 scope.go:117] "RemoveContainer" containerID="9604bbba207ba97f62775233cf4430cc951efeaa46c62a97b1db31737c2879d0"
Jul 14 20:00:32.414977 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:32.414953    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b96486789-tnn57"]
Jul 14 20:00:32.419193 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:32.419031    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7b96486789-tnn57"]
Jul 14 20:00:33.246827 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:33.246795    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b96486789-tj8km"]
Jul 14 20:00:33.247159 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:33.247148    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67ef60bd-641a-4e12-b9a5-ee3b819a19fe" containerName="cleanup"
Jul 14 20:00:33.247247 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:33.247162    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="67ef60bd-641a-4e12-b9a5-ee3b819a19fe" containerName="cleanup"
Jul 14 20:00:33.247247 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:33.247199    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95db45a8-6be8-4f3d-aae1-898ac27cb2be" containerName="manager"
Jul 14 20:00:33.247247 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:33.247209    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="95db45a8-6be8-4f3d-aae1-898ac27cb2be" containerName="manager"
Jul 14 20:00:33.247344 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:33.247295    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="67ef60bd-641a-4e12-b9a5-ee3b819a19fe" containerName="cleanup"
Jul 14 20:00:33.247344 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:33.247303    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="95db45a8-6be8-4f3d-aae1-898ac27cb2be" containerName="manager"
Jul 14 20:00:33.251662 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:33.251640    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b96486789-tj8km"
Jul 14 20:00:33.254000 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:33.253977    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 14 20:00:33.254117 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:33.253979    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-g24q9\""
Jul 14 20:00:33.254239 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:33.254223    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 14 20:00:33.259086 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:33.259061    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b96486789-tj8km"]
Jul 14 20:00:33.273683 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:33.273654    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mkx6\" (UniqueName: \"kubernetes.io/projected/838f0bc8-3271-4987-abc1-76141aea6162-kube-api-access-4mkx6\") pod \"maas-controller-7b96486789-tj8km\" (UID: \"838f0bc8-3271-4987-abc1-76141aea6162\") " pod="opendatahub/maas-controller-7b96486789-tj8km"
Jul 14 20:00:33.273788 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:33.273700    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/838f0bc8-3271-4987-abc1-76141aea6162-webhook-cert\") pod \"maas-controller-7b96486789-tj8km\" (UID: \"838f0bc8-3271-4987-abc1-76141aea6162\") " pod="opendatahub/maas-controller-7b96486789-tj8km"
Jul 14 20:00:33.375241 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:33.375207    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mkx6\" (UniqueName: \"kubernetes.io/projected/838f0bc8-3271-4987-abc1-76141aea6162-kube-api-access-4mkx6\") pod \"maas-controller-7b96486789-tj8km\" (UID: \"838f0bc8-3271-4987-abc1-76141aea6162\") " pod="opendatahub/maas-controller-7b96486789-tj8km"
Jul 14 20:00:33.375241 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:33.375255    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/838f0bc8-3271-4987-abc1-76141aea6162-webhook-cert\") pod \"maas-controller-7b96486789-tj8km\" (UID: \"838f0bc8-3271-4987-abc1-76141aea6162\") " pod="opendatahub/maas-controller-7b96486789-tj8km"
Jul 14 20:00:33.377811 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:33.377793    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/838f0bc8-3271-4987-abc1-76141aea6162-webhook-cert\") pod \"maas-controller-7b96486789-tj8km\" (UID: \"838f0bc8-3271-4987-abc1-76141aea6162\") " pod="opendatahub/maas-controller-7b96486789-tj8km"
Jul 14 20:00:33.384255 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:33.384224    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mkx6\" (UniqueName: \"kubernetes.io/projected/838f0bc8-3271-4987-abc1-76141aea6162-kube-api-access-4mkx6\") pod \"maas-controller-7b96486789-tj8km\" (UID: \"838f0bc8-3271-4987-abc1-76141aea6162\") " pod="opendatahub/maas-controller-7b96486789-tj8km"
Jul 14 20:00:33.562434 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:33.562401    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b96486789-tj8km"
Jul 14 20:00:33.697380 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:33.697357    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b96486789-tj8km"]
Jul 14 20:00:33.699822 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:00:33.699796    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838f0bc8_3271_4987_abc1_76141aea6162.slice/crio-f718afbdf862a26d3c5e6301b60b589e5ad49c4c3c0f05724d406453e55c4fcd WatchSource:0}: Error finding container f718afbdf862a26d3c5e6301b60b589e5ad49c4c3c0f05724d406453e55c4fcd: Status 404 returned error can't find the container with id f718afbdf862a26d3c5e6301b60b589e5ad49c4c3c0f05724d406453e55c4fcd
Jul 14 20:00:33.889908 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:33.889835    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95db45a8-6be8-4f3d-aae1-898ac27cb2be" path="/var/lib/kubelet/pods/95db45a8-6be8-4f3d-aae1-898ac27cb2be/volumes"
Jul 14 20:00:34.409534 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:34.409496    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b96486789-tj8km" event={"ID":"838f0bc8-3271-4987-abc1-76141aea6162","Type":"ContainerStarted","Data":"f718afbdf862a26d3c5e6301b60b589e5ad49c4c3c0f05724d406453e55c4fcd"}
Jul 14 20:00:35.415537 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:35.415502    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b96486789-tj8km" event={"ID":"838f0bc8-3271-4987-abc1-76141aea6162","Type":"ContainerStarted","Data":"222c6c43eab99ac9bfd49451be62625fc231c0fdf3e222b96455b9682273fcc5"}
Jul 14 20:00:35.415948 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:35.415606    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b96486789-tj8km"
Jul 14 20:00:35.434802 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:35.434754    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7b96486789-tj8km" podStartSLOduration=1.666821911 podStartE2EDuration="2.434734914s" totalImagesPullingTime="767.913003ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:00:35.432608578 +0000 UTC m=+788.032912297" watchObservedRunningTime="2026-07-14 20:00:35.434734914 +0000 UTC m=+788.035038616"
Jul 14 20:00:46.422270 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:00:46.422241    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b96486789-tj8km"
Jul 14 20:02:27.851457 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:02:27.851380    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tzr6_4a36af4c-9eac-465f-9a0e-d65a382530f7/ovn-acl-logging/0.log"
Jul 14 20:02:27.852454 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:02:27.852436    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tzr6_4a36af4c-9eac-465f-9a0e-d65a382530f7/ovn-acl-logging/0.log"
Jul 14 20:07:27.887094 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:07:27.887064    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tzr6_4a36af4c-9eac-465f-9a0e-d65a382530f7/ovn-acl-logging/0.log"
Jul 14 20:07:27.888949 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:07:27.888915    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tzr6_4a36af4c-9eac-465f-9a0e-d65a382530f7/ovn-acl-logging/0.log"
Jul 14 20:12:27.916104 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:27.916074    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tzr6_4a36af4c-9eac-465f-9a0e-d65a382530f7/ovn-acl-logging/0.log"
Jul 14 20:12:27.918330 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:27.918305    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tzr6_4a36af4c-9eac-465f-9a0e-d65a382530f7/ovn-acl-logging/0.log"
Jul 14 20:12:46.924337 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:46.924299    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv"]
Jul 14 20:12:46.927821 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:46.927803    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv"
Jul 14 20:12:46.930701 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:46.930680    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-84a35a9d-serving-cert\""
Jul 14 20:12:46.930787 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:46.930704    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gfkms\""
Jul 14 20:12:46.938871 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:46.938849    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv"]
Jul 14 20:12:47.029837 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:47.029808    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13924dd0-aa75-4d8f-9d36-1a98d6ce50d0-maas-api-tls\") pod \"maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv\" (UID: \"13924dd0-aa75-4d8f-9d36-1a98d6ce50d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv"
Jul 14 20:12:47.030001 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:47.029842    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q8c6\" (UniqueName: \"kubernetes.io/projected/13924dd0-aa75-4d8f-9d36-1a98d6ce50d0-kube-api-access-6q8c6\") pod \"maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv\" (UID: \"13924dd0-aa75-4d8f-9d36-1a98d6ce50d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv"
Jul 14 20:12:47.030044 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:47.030000    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13924dd0-aa75-4d8f-9d36-1a98d6ce50d0-openshift-service-ca\") pod \"maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv\" (UID: \"13924dd0-aa75-4d8f-9d36-1a98d6ce50d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv"
Jul 14 20:12:47.131198 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:47.131136    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13924dd0-aa75-4d8f-9d36-1a98d6ce50d0-openshift-service-ca\") pod \"maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv\" (UID: \"13924dd0-aa75-4d8f-9d36-1a98d6ce50d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv"
Jul 14 20:12:47.131198 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:47.131196    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13924dd0-aa75-4d8f-9d36-1a98d6ce50d0-maas-api-tls\") pod \"maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv\" (UID: \"13924dd0-aa75-4d8f-9d36-1a98d6ce50d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv"
Jul 14 20:12:47.131398 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:47.131226    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q8c6\" (UniqueName: \"kubernetes.io/projected/13924dd0-aa75-4d8f-9d36-1a98d6ce50d0-kube-api-access-6q8c6\") pod \"maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv\" (UID: \"13924dd0-aa75-4d8f-9d36-1a98d6ce50d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv"
Jul 14 20:12:47.131398 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:12:47.131356    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-serving-cert: secret "maas-api-e2e-ait-84a35a9d-serving-cert" not found
Jul 14 20:12:47.131463 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:12:47.131420    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13924dd0-aa75-4d8f-9d36-1a98d6ce50d0-maas-api-tls podName:13924dd0-aa75-4d8f-9d36-1a98d6ce50d0 nodeName:}" failed. No retries permitted until 2026-07-14 20:12:47.631402713 +0000 UTC m=+1520.231706395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/13924dd0-aa75-4d8f-9d36-1a98d6ce50d0-maas-api-tls") pod "maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv" (UID: "13924dd0-aa75-4d8f-9d36-1a98d6ce50d0") : secret "maas-api-e2e-ait-84a35a9d-serving-cert" not found
Jul 14 20:12:47.131883 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:47.131863    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13924dd0-aa75-4d8f-9d36-1a98d6ce50d0-openshift-service-ca\") pod \"maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv\" (UID: \"13924dd0-aa75-4d8f-9d36-1a98d6ce50d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv"
Jul 14 20:12:47.141209 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:47.141155    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q8c6\" (UniqueName: \"kubernetes.io/projected/13924dd0-aa75-4d8f-9d36-1a98d6ce50d0-kube-api-access-6q8c6\") pod \"maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv\" (UID: \"13924dd0-aa75-4d8f-9d36-1a98d6ce50d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv"
Jul 14 20:12:47.636029 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:47.635977    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13924dd0-aa75-4d8f-9d36-1a98d6ce50d0-maas-api-tls\") pod \"maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv\" (UID: \"13924dd0-aa75-4d8f-9d36-1a98d6ce50d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv"
Jul 14 20:12:47.638635 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:47.638604    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13924dd0-aa75-4d8f-9d36-1a98d6ce50d0-maas-api-tls\") pod \"maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv\" (UID: \"13924dd0-aa75-4d8f-9d36-1a98d6ce50d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv"
Jul 14 20:12:47.838846 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:47.838811    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv"
Jul 14 20:12:47.980165 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:47.980133    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv"]
Jul 14 20:12:47.983367 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:12:47.983328    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13924dd0_aa75_4d8f_9d36_1a98d6ce50d0.slice/crio-ea3ef9381b8692b1d591802d83b11d29e166150b6901d5501184ae653abbf15e WatchSource:0}: Error finding container ea3ef9381b8692b1d591802d83b11d29e166150b6901d5501184ae653abbf15e: Status 404 returned error can't find the container with id ea3ef9381b8692b1d591802d83b11d29e166150b6901d5501184ae653abbf15e
Jul 14 20:12:47.984668 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:47.984648    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 20:12:48.767563 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:48.767528    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv" event={"ID":"13924dd0-aa75-4d8f-9d36-1a98d6ce50d0","Type":"ContainerStarted","Data":"8836332e3aaf3b931baa743001f54c2b3bcb00eea00466d88a4eecdd6ced5532"}
Jul 14 20:12:48.767740 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:48.767579    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv" event={"ID":"13924dd0-aa75-4d8f-9d36-1a98d6ce50d0","Type":"ContainerStarted","Data":"ea3ef9381b8692b1d591802d83b11d29e166150b6901d5501184ae653abbf15e"}
Jul 14 20:12:48.767740 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:48.767662    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv"
Jul 14 20:12:48.788088 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:48.787998    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv" podStartSLOduration=2.236376402 podStartE2EDuration="2.787974618s" totalImagesPullingTime="551.598216ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:12:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:12:48.785753865 +0000 UTC m=+1521.386057568" watchObservedRunningTime="2026-07-14 20:12:48.787974618 +0000 UTC m=+1521.388278327"
Jul 14 20:12:51.995342 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:51.995305    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-vxzfh"]
Jul 14 20:12:51.998749 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:51.998731    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-vxzfh"
Jul 14 20:12:52.000950 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:52.000932    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-56w6s\""
Jul 14 20:12:52.005685 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:52.005661    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-vxzfh"]
Jul 14 20:12:52.075633 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:52.075592    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-84a35a9d-vxzfh\" (UID: \"9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-vxzfh"
Jul 14 20:12:52.176369 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:52.176333    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-84a35a9d-vxzfh\" (UID: \"9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-vxzfh"
Jul 14 20:12:52.176962 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:52.176944    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-84a35a9d-vxzfh\" (UID: \"9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-vxzfh"
Jul 14 20:12:52.310777 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:52.310745    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-vxzfh"
Jul 14 20:12:52.450985 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:52.450959    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-vxzfh"]
Jul 14 20:12:52.452713 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:12:52.452686    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0cbdd1_5eba_4f0b_95c6_a4003104a5e2.slice/crio-e35ebabf9b8c7eca0b06ce5fa5fae9b6028924e970087765d81a3c868edfef2a WatchSource:0}: Error finding container e35ebabf9b8c7eca0b06ce5fa5fae9b6028924e970087765d81a3c868edfef2a: Status 404 returned error can't find the container with id e35ebabf9b8c7eca0b06ce5fa5fae9b6028924e970087765d81a3c868edfef2a
Jul 14 20:12:52.785981 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:52.785950    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-vxzfh" event={"ID":"9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2","Type":"ContainerStarted","Data":"e35ebabf9b8c7eca0b06ce5fa5fae9b6028924e970087765d81a3c868edfef2a"}
Jul 14 20:12:53.792477 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:53.792443    2560 generic.go:356] "Generic (PLEG): container finished" podID="9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2" containerID="4d3796f2031f95fa0a79bffa631d8204295699280d7146a5c941924b3f98d92d" exitCode=7
Jul 14 20:12:53.792850 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:53.792483    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-vxzfh" event={"ID":"9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2","Type":"ContainerDied","Data":"4d3796f2031f95fa0a79bffa631d8204295699280d7146a5c941924b3f98d92d"}
Jul 14 20:12:53.792850 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:53.792653    2560 scope.go:117] "RemoveContainer" containerID="4d3796f2031f95fa0a79bffa631d8204295699280d7146a5c941924b3f98d92d"
Jul 14 20:12:54.779055 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:54.778983    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv"
Jul 14 20:12:54.798840 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:54.798807    2560 generic.go:356] "Generic (PLEG): container finished" podID="9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2" containerID="6f422f56c5c0da4702d3c92ed51eb8793ae7f4bebf022662b90952d7ea25201b" exitCode=7
Jul 14 20:12:54.799240 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:54.798882    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-vxzfh" event={"ID":"9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2","Type":"ContainerDied","Data":"6f422f56c5c0da4702d3c92ed51eb8793ae7f4bebf022662b90952d7ea25201b"}
Jul 14 20:12:54.799240 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:54.798935    2560 scope.go:117] "RemoveContainer" containerID="4d3796f2031f95fa0a79bffa631d8204295699280d7146a5c941924b3f98d92d"
Jul 14 20:12:54.799240 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:54.799140    2560 scope.go:117] "RemoveContainer" containerID="6f422f56c5c0da4702d3c92ed51eb8793ae7f4bebf022662b90952d7ea25201b"
Jul 14 20:12:54.799422 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:12:54.799402    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-84a35a9d-vxzfh_odh-ai-gateway-infra(9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-vxzfh" podUID="9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2"
Jul 14 20:12:55.805323 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:12:55.805290    2560 scope.go:117] "RemoveContainer" containerID="6f422f56c5c0da4702d3c92ed51eb8793ae7f4bebf022662b90952d7ea25201b"
Jul 14 20:12:55.805766 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:12:55.805531    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-84a35a9d-vxzfh_odh-ai-gateway-infra(9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-vxzfh" podUID="9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2"
Jul 14 20:13:08.884065 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:08.884034    2560 scope.go:117] "RemoveContainer" containerID="6f422f56c5c0da4702d3c92ed51eb8793ae7f4bebf022662b90952d7ea25201b"
Jul 14 20:13:09.870038 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:09.869961    2560 generic.go:356] "Generic (PLEG): container finished" podID="9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2" containerID="05c3bc90990df336b8c557032ae294b395ce8d40e90adce1412c662f8bea7e1d" exitCode=0
Jul 14 20:13:09.870199 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:09.870034    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-vxzfh" event={"ID":"9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2","Type":"ContainerDied","Data":"05c3bc90990df336b8c557032ae294b395ce8d40e90adce1412c662f8bea7e1d"}
Jul 14 20:13:09.870199 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:09.870093    2560 scope.go:117] "RemoveContainer" containerID="6f422f56c5c0da4702d3c92ed51eb8793ae7f4bebf022662b90952d7ea25201b"
Jul 14 20:13:10.896023 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:10.895984    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-vxzfh"]
Jul 14 20:13:11.016274 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:11.016240    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-vxzfh"
Jul 14 20:13:11.056245 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:11.056205    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2-maas-api-service-ca\") pod \"9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2\" (UID: \"9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2\") "
Jul 14 20:13:11.056559 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:11.056533    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2" (UID: "9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 20:13:11.157134 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:11.157034    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2-maas-api-service-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:13:11.883786 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:11.883761    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-vxzfh"
Jul 14 20:13:11.889883 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:11.889848    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-vxzfh" event={"ID":"9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2","Type":"ContainerDied","Data":"e35ebabf9b8c7eca0b06ce5fa5fae9b6028924e970087765d81a3c868edfef2a"}
Jul 14 20:13:11.890014 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:11.889907    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e35ebabf9b8c7eca0b06ce5fa5fae9b6028924e970087765d81a3c868edfef2a"
Jul 14 20:13:11.915310 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:11.915280    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-vxzfh"]
Jul 14 20:13:11.917018 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:11.916999    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-vxzfh"]
Jul 14 20:13:13.889752 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:13.889716    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2" path="/var/lib/kubelet/pods/9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2/volumes"
Jul 14 20:13:42.036604 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:42.036565    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-d9swv"]
Jul 14 20:13:42.037078 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:42.036951    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2" containerName="revoke-keys"
Jul 14 20:13:42.037078 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:42.036963    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2" containerName="revoke-keys"
Jul 14 20:13:42.037078 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:42.036971    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2" containerName="revoke-keys"
Jul 14 20:13:42.037078 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:42.036977    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2" containerName="revoke-keys"
Jul 14 20:13:42.037078 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:42.036989    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2" containerName="revoke-keys"
Jul 14 20:13:42.037078 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:42.036994    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2" containerName="revoke-keys"
Jul 14 20:13:42.037078 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:42.037048    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2" containerName="revoke-keys"
Jul 14 20:13:42.037078 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:42.037057    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2" containerName="revoke-keys"
Jul 14 20:13:42.040320 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:42.040304    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-d9swv"
Jul 14 20:13:42.042645 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:42.042622    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-56w6s\""
Jul 14 20:13:42.053961 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:42.053935    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-d9swv"]
Jul 14 20:13:42.139597 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:42.139559    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc85368-987f-4c9f-986e-85b0f0b283fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-84a35a9d-d9swv\" (UID: \"fdc85368-987f-4c9f-986e-85b0f0b283fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-d9swv"
Jul 14 20:13:42.240409 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:42.240371    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc85368-987f-4c9f-986e-85b0f0b283fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-84a35a9d-d9swv\" (UID: \"fdc85368-987f-4c9f-986e-85b0f0b283fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-d9swv"
Jul 14 20:13:42.241009 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:42.240987    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc85368-987f-4c9f-986e-85b0f0b283fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-84a35a9d-d9swv\" (UID: \"fdc85368-987f-4c9f-986e-85b0f0b283fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-d9swv"
Jul 14 20:13:42.350583 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:42.350502    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-d9swv"
Jul 14 20:13:42.484585 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:42.484526    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-d9swv"]
Jul 14 20:13:42.486798 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:13:42.486772    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc85368_987f_4c9f_986e_85b0f0b283fb.slice/crio-a4e066d4af83607b99c249c58ee499197c873bdd4a04f6cf194e4d9a6589ecea WatchSource:0}: Error finding container a4e066d4af83607b99c249c58ee499197c873bdd4a04f6cf194e4d9a6589ecea: Status 404 returned error can't find the container with id a4e066d4af83607b99c249c58ee499197c873bdd4a04f6cf194e4d9a6589ecea
Jul 14 20:13:43.032477 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:43.032439    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-d9swv" event={"ID":"fdc85368-987f-4c9f-986e-85b0f0b283fb","Type":"ContainerStarted","Data":"a8ca8d23db22a6b1c0975987d1b57fd8cd13be7e7c4f431769842ab4048f16c8"}
Jul 14 20:13:43.032660 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:43.032520    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-d9swv" event={"ID":"fdc85368-987f-4c9f-986e-85b0f0b283fb","Type":"ContainerStarted","Data":"a4e066d4af83607b99c249c58ee499197c873bdd4a04f6cf194e4d9a6589ecea"}
Jul 14 20:13:43.050259 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:43.050204    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-d9swv" podStartSLOduration=1.050188622 podStartE2EDuration="1.050188622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:13:43.047705717 +0000 UTC m=+1575.648009421" watchObservedRunningTime="2026-07-14 20:13:43.050188622 +0000 UTC m=+1575.650492315"
Jul 14 20:13:44.038881 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:44.038851    2560 generic.go:356] "Generic (PLEG): container finished" podID="fdc85368-987f-4c9f-986e-85b0f0b283fb" containerID="a8ca8d23db22a6b1c0975987d1b57fd8cd13be7e7c4f431769842ab4048f16c8" exitCode=0
Jul 14 20:13:44.039055 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:44.038949    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-d9swv" event={"ID":"fdc85368-987f-4c9f-986e-85b0f0b283fb","Type":"ContainerDied","Data":"a8ca8d23db22a6b1c0975987d1b57fd8cd13be7e7c4f431769842ab4048f16c8"}
Jul 14 20:13:45.174560 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:45.174532    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-d9swv"
Jul 14 20:13:45.269193 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:45.269140    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc85368-987f-4c9f-986e-85b0f0b283fb-maas-api-service-ca\") pod \"fdc85368-987f-4c9f-986e-85b0f0b283fb\" (UID: \"fdc85368-987f-4c9f-986e-85b0f0b283fb\") "
Jul 14 20:13:45.269551 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:45.269525    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc85368-987f-4c9f-986e-85b0f0b283fb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fdc85368-987f-4c9f-986e-85b0f0b283fb" (UID: "fdc85368-987f-4c9f-986e-85b0f0b283fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 20:13:45.370350 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:45.370194    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc85368-987f-4c9f-986e-85b0f0b283fb-maas-api-service-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:13:46.049620 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:46.049590    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-d9swv"
Jul 14 20:13:46.049783 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:46.049593    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84a35a9d-d9swv" event={"ID":"fdc85368-987f-4c9f-986e-85b0f0b283fb","Type":"ContainerDied","Data":"a4e066d4af83607b99c249c58ee499197c873bdd4a04f6cf194e4d9a6589ecea"}
Jul 14 20:13:46.049783 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:46.049711    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4e066d4af83607b99c249c58ee499197c873bdd4a04f6cf194e4d9a6589ecea"
Jul 14 20:13:52.464036 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:52.463983    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv"]
Jul 14 20:13:52.464589 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:52.464248    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv" podUID="13924dd0-aa75-4d8f-9d36-1a98d6ce50d0" containerName="maas-api" containerID="cri-o://8836332e3aaf3b931baa743001f54c2b3bcb00eea00466d88a4eecdd6ced5532" gracePeriod=30
Jul 14 20:13:52.717044 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:52.716980    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv"
Jul 14 20:13:52.730365 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:52.730341    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13924dd0-aa75-4d8f-9d36-1a98d6ce50d0-maas-api-tls\") pod \"13924dd0-aa75-4d8f-9d36-1a98d6ce50d0\" (UID: \"13924dd0-aa75-4d8f-9d36-1a98d6ce50d0\") "
Jul 14 20:13:52.730529 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:52.730373    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6q8c6\" (UniqueName: \"kubernetes.io/projected/13924dd0-aa75-4d8f-9d36-1a98d6ce50d0-kube-api-access-6q8c6\") pod \"13924dd0-aa75-4d8f-9d36-1a98d6ce50d0\" (UID: \"13924dd0-aa75-4d8f-9d36-1a98d6ce50d0\") "
Jul 14 20:13:52.730529 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:52.730469    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13924dd0-aa75-4d8f-9d36-1a98d6ce50d0-openshift-service-ca\") pod \"13924dd0-aa75-4d8f-9d36-1a98d6ce50d0\" (UID: \"13924dd0-aa75-4d8f-9d36-1a98d6ce50d0\") "
Jul 14 20:13:52.730785 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:52.730757    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13924dd0-aa75-4d8f-9d36-1a98d6ce50d0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "13924dd0-aa75-4d8f-9d36-1a98d6ce50d0" (UID: "13924dd0-aa75-4d8f-9d36-1a98d6ce50d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 20:13:52.732700 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:52.732673    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13924dd0-aa75-4d8f-9d36-1a98d6ce50d0-kube-api-access-6q8c6" (OuterVolumeSpecName: "kube-api-access-6q8c6") pod "13924dd0-aa75-4d8f-9d36-1a98d6ce50d0" (UID: "13924dd0-aa75-4d8f-9d36-1a98d6ce50d0"). InnerVolumeSpecName "kube-api-access-6q8c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 20:13:52.732768 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:52.732707    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13924dd0-aa75-4d8f-9d36-1a98d6ce50d0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "13924dd0-aa75-4d8f-9d36-1a98d6ce50d0" (UID: "13924dd0-aa75-4d8f-9d36-1a98d6ce50d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 20:13:52.831380 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:52.831334    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13924dd0-aa75-4d8f-9d36-1a98d6ce50d0-maas-api-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:13:52.831380 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:52.831375    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q8c6\" (UniqueName: \"kubernetes.io/projected/13924dd0-aa75-4d8f-9d36-1a98d6ce50d0-kube-api-access-6q8c6\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:13:52.831380 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:52.831384    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13924dd0-aa75-4d8f-9d36-1a98d6ce50d0-openshift-service-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:13:53.085160 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:53.085118    2560 generic.go:356] "Generic (PLEG): container finished" podID="13924dd0-aa75-4d8f-9d36-1a98d6ce50d0" containerID="8836332e3aaf3b931baa743001f54c2b3bcb00eea00466d88a4eecdd6ced5532" exitCode=0
Jul 14 20:13:53.085354 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:53.085197    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv"
Jul 14 20:13:53.085354 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:53.085218    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv" event={"ID":"13924dd0-aa75-4d8f-9d36-1a98d6ce50d0","Type":"ContainerDied","Data":"8836332e3aaf3b931baa743001f54c2b3bcb00eea00466d88a4eecdd6ced5532"}
Jul 14 20:13:53.085354 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:53.085278    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv" event={"ID":"13924dd0-aa75-4d8f-9d36-1a98d6ce50d0","Type":"ContainerDied","Data":"ea3ef9381b8692b1d591802d83b11d29e166150b6901d5501184ae653abbf15e"}
Jul 14 20:13:53.085354 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:53.085301    2560 scope.go:117] "RemoveContainer" containerID="8836332e3aaf3b931baa743001f54c2b3bcb00eea00466d88a4eecdd6ced5532"
Jul 14 20:13:53.095360 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:53.095339    2560 scope.go:117] "RemoveContainer" containerID="8836332e3aaf3b931baa743001f54c2b3bcb00eea00466d88a4eecdd6ced5532"
Jul 14 20:13:53.095662 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:13:53.095640    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8836332e3aaf3b931baa743001f54c2b3bcb00eea00466d88a4eecdd6ced5532\": container with ID starting with 8836332e3aaf3b931baa743001f54c2b3bcb00eea00466d88a4eecdd6ced5532 not found: ID does not exist" containerID="8836332e3aaf3b931baa743001f54c2b3bcb00eea00466d88a4eecdd6ced5532"
Jul 14 20:13:53.095713 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:53.095672    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8836332e3aaf3b931baa743001f54c2b3bcb00eea00466d88a4eecdd6ced5532"} err="failed to get container status \"8836332e3aaf3b931baa743001f54c2b3bcb00eea00466d88a4eecdd6ced5532\": rpc error: code = NotFound desc = could not find container \"8836332e3aaf3b931baa743001f54c2b3bcb00eea00466d88a4eecdd6ced5532\": container with ID starting with 8836332e3aaf3b931baa743001f54c2b3bcb00eea00466d88a4eecdd6ced5532 not found: ID does not exist"
Jul 14 20:13:53.107095 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:53.107071    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv"]
Jul 14 20:13:53.111107 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:53.111085    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84a35a9d-dd4b9fd4-sw9gv"]
Jul 14 20:13:53.889740 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:13:53.889711    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13924dd0-aa75-4d8f-9d36-1a98d6ce50d0" path="/var/lib/kubelet/pods/13924dd0-aa75-4d8f-9d36-1a98d6ce50d0/volumes"
Jul 14 20:14:09.273792 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:09.273752    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt"]
Jul 14 20:14:09.274436 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:09.274411    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13924dd0-aa75-4d8f-9d36-1a98d6ce50d0" containerName="maas-api"
Jul 14 20:14:09.274436 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:09.274436    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="13924dd0-aa75-4d8f-9d36-1a98d6ce50d0" containerName="maas-api"
Jul 14 20:14:09.274641 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:09.274452    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdc85368-987f-4c9f-986e-85b0f0b283fb" containerName="revoke-keys"
Jul 14 20:14:09.274641 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:09.274460    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdc85368-987f-4c9f-986e-85b0f0b283fb" containerName="revoke-keys"
Jul 14 20:14:09.274641 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:09.274587    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="13924dd0-aa75-4d8f-9d36-1a98d6ce50d0" containerName="maas-api"
Jul 14 20:14:09.274641 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:09.274600    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c0cbdd1-5eba-4f0b-95c6-a4003104a5e2" containerName="revoke-keys"
Jul 14 20:14:09.274641 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:09.274607    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdc85368-987f-4c9f-986e-85b0f0b283fb" containerName="revoke-keys"
Jul 14 20:14:09.277719 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:09.277701    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt"
Jul 14 20:14:09.279985 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:09.279964    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-9ead5203-serving-cert\""
Jul 14 20:14:09.280101 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:09.280044    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gfkms\""
Jul 14 20:14:09.290077 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:09.290057    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt"]
Jul 14 20:14:09.378016 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:09.377981    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70c4887-f333-45fe-b5e3-641df4c815e2-openshift-service-ca\") pod \"maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt\" (UID: \"f70c4887-f333-45fe-b5e3-641df4c815e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt"
Jul 14 20:14:09.378243 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:09.378052    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz5bq\" (UniqueName: \"kubernetes.io/projected/f70c4887-f333-45fe-b5e3-641df4c815e2-kube-api-access-sz5bq\") pod \"maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt\" (UID: \"f70c4887-f333-45fe-b5e3-641df4c815e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt"
Jul 14 20:14:09.378243 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:09.378109    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f70c4887-f333-45fe-b5e3-641df4c815e2-maas-api-tls\") pod \"maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt\" (UID: \"f70c4887-f333-45fe-b5e3-641df4c815e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt"
Jul 14 20:14:09.478830 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:09.478796    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70c4887-f333-45fe-b5e3-641df4c815e2-openshift-service-ca\") pod \"maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt\" (UID: \"f70c4887-f333-45fe-b5e3-641df4c815e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt"
Jul 14 20:14:09.479009 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:09.478871    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz5bq\" (UniqueName: \"kubernetes.io/projected/f70c4887-f333-45fe-b5e3-641df4c815e2-kube-api-access-sz5bq\") pod \"maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt\" (UID: \"f70c4887-f333-45fe-b5e3-641df4c815e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt"
Jul 14 20:14:09.479009 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:09.478945    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f70c4887-f333-45fe-b5e3-641df4c815e2-maas-api-tls\") pod \"maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt\" (UID: \"f70c4887-f333-45fe-b5e3-641df4c815e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt"
Jul 14 20:14:09.479116 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:14:09.479098    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-serving-cert: secret "maas-api-e2e-migrate-9ead5203-serving-cert" not found
Jul 14 20:14:09.479166 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:14:09.479157    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f70c4887-f333-45fe-b5e3-641df4c815e2-maas-api-tls podName:f70c4887-f333-45fe-b5e3-641df4c815e2 nodeName:}" failed. No retries permitted until 2026-07-14 20:14:09.979141945 +0000 UTC m=+1602.579445627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f70c4887-f333-45fe-b5e3-641df4c815e2-maas-api-tls") pod "maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt" (UID: "f70c4887-f333-45fe-b5e3-641df4c815e2") : secret "maas-api-e2e-migrate-9ead5203-serving-cert" not found
Jul 14 20:14:09.479455 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:09.479437    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70c4887-f333-45fe-b5e3-641df4c815e2-openshift-service-ca\") pod \"maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt\" (UID: \"f70c4887-f333-45fe-b5e3-641df4c815e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt"
Jul 14 20:14:09.487887 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:09.487858    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz5bq\" (UniqueName: \"kubernetes.io/projected/f70c4887-f333-45fe-b5e3-641df4c815e2-kube-api-access-sz5bq\") pod \"maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt\" (UID: \"f70c4887-f333-45fe-b5e3-641df4c815e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt"
Jul 14 20:14:09.984657 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:09.984614    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f70c4887-f333-45fe-b5e3-641df4c815e2-maas-api-tls\") pod \"maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt\" (UID: \"f70c4887-f333-45fe-b5e3-641df4c815e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt"
Jul 14 20:14:09.987285 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:09.987255    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f70c4887-f333-45fe-b5e3-641df4c815e2-maas-api-tls\") pod \"maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt\" (UID: \"f70c4887-f333-45fe-b5e3-641df4c815e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt"
Jul 14 20:14:10.188529 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:10.188491    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt"
Jul 14 20:14:10.329323 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:10.329296    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt"]
Jul 14 20:14:10.331795 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:14:10.331763    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70c4887_f333_45fe_b5e3_641df4c815e2.slice/crio-e8e4dc3a5520c469534ef058057875121d7c23f2c20d2e233051375625870bf1 WatchSource:0}: Error finding container e8e4dc3a5520c469534ef058057875121d7c23f2c20d2e233051375625870bf1: Status 404 returned error can't find the container with id e8e4dc3a5520c469534ef058057875121d7c23f2c20d2e233051375625870bf1
Jul 14 20:14:11.177695 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:11.177662    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt" event={"ID":"f70c4887-f333-45fe-b5e3-641df4c815e2","Type":"ContainerStarted","Data":"cf090f584d4807c2d5b5a32a7df89594575edd50312df71ec7a084d7d052bc3d"}
Jul 14 20:14:11.177880 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:11.177711    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt" event={"ID":"f70c4887-f333-45fe-b5e3-641df4c815e2","Type":"ContainerStarted","Data":"e8e4dc3a5520c469534ef058057875121d7c23f2c20d2e233051375625870bf1"}
Jul 14 20:14:11.177880 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:11.177757    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt"
Jul 14 20:14:11.196603 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:11.196551    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt" podStartSLOduration=1.7700113690000001 podStartE2EDuration="2.196532354s" totalImagesPullingTime="426.520985ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:14:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:14:11.196445007 +0000 UTC m=+1603.796748722" watchObservedRunningTime="2026-07-14 20:14:11.196532354 +0000 UTC m=+1603.796836058"
Jul 14 20:14:13.755107 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:13.755063    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-5ngcc"]
Jul 14 20:14:13.758530 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:13.758513    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-5ngcc"
Jul 14 20:14:13.760708 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:13.760689    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-56w6s\""
Jul 14 20:14:13.766645 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:13.766620    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-5ngcc"]
Jul 14 20:14:13.821243 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:13.821203    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2309fd55-26b4-40e2-934c-33b9f0231e21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9ead5203-5ngcc\" (UID: \"2309fd55-26b4-40e2-934c-33b9f0231e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-5ngcc"
Jul 14 20:14:13.921917 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:13.921829    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2309fd55-26b4-40e2-934c-33b9f0231e21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9ead5203-5ngcc\" (UID: \"2309fd55-26b4-40e2-934c-33b9f0231e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-5ngcc"
Jul 14 20:14:13.922500 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:13.922479    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2309fd55-26b4-40e2-934c-33b9f0231e21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9ead5203-5ngcc\" (UID: \"2309fd55-26b4-40e2-934c-33b9f0231e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-5ngcc"
Jul 14 20:14:14.069614 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:14.069581    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-5ngcc"
Jul 14 20:14:14.202657 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:14.202634    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-5ngcc"]
Jul 14 20:14:14.205101 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:14:14.205073    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2309fd55_26b4_40e2_934c_33b9f0231e21.slice/crio-c2bb89df2b841496a5d7c4b9f9df5f6a68dc604c8c692997bbd40eac3a5870f4 WatchSource:0}: Error finding container c2bb89df2b841496a5d7c4b9f9df5f6a68dc604c8c692997bbd40eac3a5870f4: Status 404 returned error can't find the container with id c2bb89df2b841496a5d7c4b9f9df5f6a68dc604c8c692997bbd40eac3a5870f4
Jul 14 20:14:15.198302 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:15.198223    2560 generic.go:356] "Generic (PLEG): container finished" podID="2309fd55-26b4-40e2-934c-33b9f0231e21" containerID="49696c34882808a4abd55457c0bb53b58ddf18d6a891587d9203c341d075f640" exitCode=7
Jul 14 20:14:15.198302 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:15.198288    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-5ngcc" event={"ID":"2309fd55-26b4-40e2-934c-33b9f0231e21","Type":"ContainerDied","Data":"49696c34882808a4abd55457c0bb53b58ddf18d6a891587d9203c341d075f640"}
Jul 14 20:14:15.198672 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:15.198324    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-5ngcc" event={"ID":"2309fd55-26b4-40e2-934c-33b9f0231e21","Type":"ContainerStarted","Data":"c2bb89df2b841496a5d7c4b9f9df5f6a68dc604c8c692997bbd40eac3a5870f4"}
Jul 14 20:14:15.198672 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:15.198470    2560 scope.go:117] "RemoveContainer" containerID="49696c34882808a4abd55457c0bb53b58ddf18d6a891587d9203c341d075f640"
Jul 14 20:14:16.204476 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:16.204388    2560 generic.go:356] "Generic (PLEG): container finished" podID="2309fd55-26b4-40e2-934c-33b9f0231e21" containerID="11b184de7e9b9d2825b54f5a948efc28037f3ea2809ad46185385a51ead65c6f" exitCode=7
Jul 14 20:14:16.204949 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:16.204482    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-5ngcc" event={"ID":"2309fd55-26b4-40e2-934c-33b9f0231e21","Type":"ContainerDied","Data":"11b184de7e9b9d2825b54f5a948efc28037f3ea2809ad46185385a51ead65c6f"}
Jul 14 20:14:16.204949 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:16.204553    2560 scope.go:117] "RemoveContainer" containerID="49696c34882808a4abd55457c0bb53b58ddf18d6a891587d9203c341d075f640"
Jul 14 20:14:16.204949 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:16.204653    2560 scope.go:117] "RemoveContainer" containerID="11b184de7e9b9d2825b54f5a948efc28037f3ea2809ad46185385a51ead65c6f"
Jul 14 20:14:16.204949 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:14:16.204897    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-9ead5203-5ngcc_odh-ai-gateway-infra(2309fd55-26b4-40e2-934c-33b9f0231e21)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-5ngcc" podUID="2309fd55-26b4-40e2-934c-33b9f0231e21"
Jul 14 20:14:17.190089 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:17.190055    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt"
Jul 14 20:14:17.211604 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:17.211580    2560 scope.go:117] "RemoveContainer" containerID="11b184de7e9b9d2825b54f5a948efc28037f3ea2809ad46185385a51ead65c6f"
Jul 14 20:14:17.211957 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:14:17.211776    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-9ead5203-5ngcc_odh-ai-gateway-infra(2309fd55-26b4-40e2-934c-33b9f0231e21)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-5ngcc" podUID="2309fd55-26b4-40e2-934c-33b9f0231e21"
Jul 14 20:14:28.884082 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:28.884050    2560 scope.go:117] "RemoveContainer" containerID="11b184de7e9b9d2825b54f5a948efc28037f3ea2809ad46185385a51ead65c6f"
Jul 14 20:14:29.280258 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:29.280222    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-5ngcc" event={"ID":"2309fd55-26b4-40e2-934c-33b9f0231e21","Type":"ContainerStarted","Data":"6ef449b70a3e853ec23781a54f3726082e85f96d21c6c95c9247b9ac23591a4f"}
Jul 14 20:14:29.298830 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:29.298784    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-5ngcc" podStartSLOduration=16.298769078 podStartE2EDuration="16.298769078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:14:29.297265313 +0000 UTC m=+1621.897569016" watchObservedRunningTime="2026-07-14 20:14:29.298769078 +0000 UTC m=+1621.899072781"
Jul 14 20:14:30.286388 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:30.286356    2560 generic.go:356] "Generic (PLEG): container finished" podID="2309fd55-26b4-40e2-934c-33b9f0231e21" containerID="6ef449b70a3e853ec23781a54f3726082e85f96d21c6c95c9247b9ac23591a4f" exitCode=0
Jul 14 20:14:30.286764 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:30.286432    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-5ngcc" event={"ID":"2309fd55-26b4-40e2-934c-33b9f0231e21","Type":"ContainerDied","Data":"6ef449b70a3e853ec23781a54f3726082e85f96d21c6c95c9247b9ac23591a4f"}
Jul 14 20:14:30.286764 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:30.286492    2560 scope.go:117] "RemoveContainer" containerID="11b184de7e9b9d2825b54f5a948efc28037f3ea2809ad46185385a51ead65c6f"
Jul 14 20:14:30.314014 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:30.313981    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-5ngcc"]
Jul 14 20:14:31.427654 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:31.427633    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-5ngcc"
Jul 14 20:14:31.589576 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:31.589491    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2309fd55-26b4-40e2-934c-33b9f0231e21-maas-api-service-ca\") pod \"2309fd55-26b4-40e2-934c-33b9f0231e21\" (UID: \"2309fd55-26b4-40e2-934c-33b9f0231e21\") "
Jul 14 20:14:31.589845 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:31.589822    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2309fd55-26b4-40e2-934c-33b9f0231e21-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2309fd55-26b4-40e2-934c-33b9f0231e21" (UID: "2309fd55-26b4-40e2-934c-33b9f0231e21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 20:14:31.690084 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:31.690053    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2309fd55-26b4-40e2-934c-33b9f0231e21-maas-api-service-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:14:32.299950 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:32.299915    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-5ngcc" event={"ID":"2309fd55-26b4-40e2-934c-33b9f0231e21","Type":"ContainerDied","Data":"c2bb89df2b841496a5d7c4b9f9df5f6a68dc604c8c692997bbd40eac3a5870f4"}
Jul 14 20:14:32.300159 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:32.299981    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2bb89df2b841496a5d7c4b9f9df5f6a68dc604c8c692997bbd40eac3a5870f4"
Jul 14 20:14:32.300159 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:32.299923    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-5ngcc"
Jul 14 20:14:32.330106 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:32.330076    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-5ngcc"]
Jul 14 20:14:32.343574 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:32.343550    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-5ngcc"]
Jul 14 20:14:33.889640 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:14:33.889602    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2309fd55-26b4-40e2-934c-33b9f0231e21" path="/var/lib/kubelet/pods/2309fd55-26b4-40e2-934c-33b9f0231e21/volumes"
Jul 14 20:15:00.147760 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:00.147727    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-9ead5203-29734335-t42fq"]
Jul 14 20:15:00.148242 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:00.148105    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2309fd55-26b4-40e2-934c-33b9f0231e21" containerName="revoke-keys"
Jul 14 20:15:00.148242 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:00.148117    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2309fd55-26b4-40e2-934c-33b9f0231e21" containerName="revoke-keys"
Jul 14 20:15:00.148242 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:00.148152    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2309fd55-26b4-40e2-934c-33b9f0231e21" containerName="revoke-keys"
Jul 14 20:15:00.148242 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:00.148158    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2309fd55-26b4-40e2-934c-33b9f0231e21" containerName="revoke-keys"
Jul 14 20:15:00.148242 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:00.148231    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2309fd55-26b4-40e2-934c-33b9f0231e21" containerName="revoke-keys"
Jul 14 20:15:00.148242 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:00.148239    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2309fd55-26b4-40e2-934c-33b9f0231e21" containerName="revoke-keys"
Jul 14 20:15:00.150923 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:00.150899    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-9ead5203-29734335-t42fq"
Jul 14 20:15:00.151558 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:00.151533    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734335-fkdrw"]
Jul 14 20:15:00.151884 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:00.151868    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2309fd55-26b4-40e2-934c-33b9f0231e21" containerName="revoke-keys"
Jul 14 20:15:00.151884 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:00.151881    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2309fd55-26b4-40e2-934c-33b9f0231e21" containerName="revoke-keys"
Jul 14 20:15:00.152014 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:00.151943    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2309fd55-26b4-40e2-934c-33b9f0231e21" containerName="revoke-keys"
Jul 14 20:15:00.153261 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:00.153243    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-56w6s\""
Jul 14 20:15:00.154904 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:00.154886    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734335-fkdrw"
Jul 14 20:15:00.160680 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:00.160658    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-9ead5203-29734335-t42fq"]
Jul 14 20:15:00.165211 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:00.165167    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-9ead5203-29734335-t42fq"
Jul 14 20:15:00.166861 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:00.166836    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734335-fkdrw"]
Jul 14 20:15:00.172147 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:00.172128    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734335-fkdrw"
Jul 14 20:15:00.317846 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:00.317810    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-9ead5203-29734335-t42fq"]
Jul 14 20:15:00.322505 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:15:00.322475    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99f1449_dd77_461d_98b5_58834ee50b06.slice/crio-35bec30137074357a0349ee43ad1070f91666334c88d6754a4c5a98e7431ec4d WatchSource:0}: Error finding container 35bec30137074357a0349ee43ad1070f91666334c88d6754a4c5a98e7431ec4d: Status 404 returned error can't find the container with id 35bec30137074357a0349ee43ad1070f91666334c88d6754a4c5a98e7431ec4d
Jul 14 20:15:00.343711 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:00.343639    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734335-fkdrw"]
Jul 14 20:15:00.346291 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:15:00.346263    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1206f397_6723_4384_8e79_304508cbce93.slice/crio-ad58c83b8b1846790066a377bf1113a9797116f178034583aa463375e9fa2ff0 WatchSource:0}: Error finding container ad58c83b8b1846790066a377bf1113a9797116f178034583aa463375e9fa2ff0: Status 404 returned error can't find the container with id ad58c83b8b1846790066a377bf1113a9797116f178034583aa463375e9fa2ff0
Jul 14 20:15:00.443089 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:00.442996    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-9ead5203-29734335-t42fq" event={"ID":"e99f1449-dd77-461d-98b5-58834ee50b06","Type":"ContainerStarted","Data":"35bec30137074357a0349ee43ad1070f91666334c88d6754a4c5a98e7431ec4d"}
Jul 14 20:15:00.444184 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:00.444147    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734335-fkdrw" event={"ID":"1206f397-6723-4384-8e79-304508cbce93","Type":"ContainerStarted","Data":"ad58c83b8b1846790066a377bf1113a9797116f178034583aa463375e9fa2ff0"}
Jul 14 20:15:01.450999 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:01.450911    2560 generic.go:356] "Generic (PLEG): container finished" podID="1206f397-6723-4384-8e79-304508cbce93" containerID="1b72c9de18f27444f7d33f91a453f76fb27dd1e4b08affc98aedd062681ba6ab" exitCode=0
Jul 14 20:15:01.451508 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:01.451012    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734335-fkdrw" event={"ID":"1206f397-6723-4384-8e79-304508cbce93","Type":"ContainerDied","Data":"1b72c9de18f27444f7d33f91a453f76fb27dd1e4b08affc98aedd062681ba6ab"}
Jul 14 20:15:01.452630 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:01.452606    2560 generic.go:356] "Generic (PLEG): container finished" podID="e99f1449-dd77-461d-98b5-58834ee50b06" containerID="78e129ebd27a543c15c2254f920af8a172852ccd24632195f40762709e84eb3a" exitCode=0
Jul 14 20:15:01.452732 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:01.452644    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-9ead5203-29734335-t42fq" event={"ID":"e99f1449-dd77-461d-98b5-58834ee50b06","Type":"ContainerDied","Data":"78e129ebd27a543c15c2254f920af8a172852ccd24632195f40762709e84eb3a"}
Jul 14 20:15:02.630272 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:02.630249    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-9ead5203-29734335-t42fq"
Jul 14 20:15:02.633154 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:02.633138    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734335-fkdrw"
Jul 14 20:15:03.464335 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:03.464301    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-9ead5203-29734335-t42fq"
Jul 14 20:15:03.464547 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:03.464294    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-9ead5203-29734335-t42fq" event={"ID":"e99f1449-dd77-461d-98b5-58834ee50b06","Type":"ContainerDied","Data":"35bec30137074357a0349ee43ad1070f91666334c88d6754a4c5a98e7431ec4d"}
Jul 14 20:15:03.464547 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:03.464491    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35bec30137074357a0349ee43ad1070f91666334c88d6754a4c5a98e7431ec4d"
Jul 14 20:15:03.465578 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:03.465559    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734335-fkdrw"
Jul 14 20:15:03.465677 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:03.465585    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734335-fkdrw" event={"ID":"1206f397-6723-4384-8e79-304508cbce93","Type":"ContainerDied","Data":"ad58c83b8b1846790066a377bf1113a9797116f178034583aa463375e9fa2ff0"}
Jul 14 20:15:03.465677 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:03.465620    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad58c83b8b1846790066a377bf1113a9797116f178034583aa463375e9fa2ff0"
Jul 14 20:15:03.808527 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:03.808489    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-9sdrj"]
Jul 14 20:15:03.808912 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:03.808895    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1206f397-6723-4384-8e79-304508cbce93" containerName="cleanup"
Jul 14 20:15:03.808912 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:03.808907    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1206f397-6723-4384-8e79-304508cbce93" containerName="cleanup"
Jul 14 20:15:03.808985 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:03.808925    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e99f1449-dd77-461d-98b5-58834ee50b06" containerName="cleanup"
Jul 14 20:15:03.808985 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:03.808930    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e99f1449-dd77-461d-98b5-58834ee50b06" containerName="cleanup"
Jul 14 20:15:03.809073 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:03.808990    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e99f1449-dd77-461d-98b5-58834ee50b06" containerName="cleanup"
Jul 14 20:15:03.809073 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:03.808997    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1206f397-6723-4384-8e79-304508cbce93" containerName="cleanup"
Jul 14 20:15:03.812029 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:03.812011    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-9sdrj"
Jul 14 20:15:03.814568 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:03.814548    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-56w6s\""
Jul 14 20:15:03.821349 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:03.821296    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-9sdrj"]
Jul 14 20:15:03.982658 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:03.982333    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac4df9d-fe96-4dfa-87dc-7d36588dda49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9ead5203-9sdrj\" (UID: \"cac4df9d-fe96-4dfa-87dc-7d36588dda49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-9sdrj"
Jul 14 20:15:04.083382 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:04.083284    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac4df9d-fe96-4dfa-87dc-7d36588dda49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9ead5203-9sdrj\" (UID: \"cac4df9d-fe96-4dfa-87dc-7d36588dda49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-9sdrj"
Jul 14 20:15:04.083912 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:04.083892    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac4df9d-fe96-4dfa-87dc-7d36588dda49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9ead5203-9sdrj\" (UID: \"cac4df9d-fe96-4dfa-87dc-7d36588dda49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-9sdrj"
Jul 14 20:15:04.123544 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:04.123510    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-9sdrj"
Jul 14 20:15:04.260451 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:04.260423    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-9sdrj"]
Jul 14 20:15:04.262821 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:15:04.262791    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac4df9d_fe96_4dfa_87dc_7d36588dda49.slice/crio-b6a49c6bd26c495f2282a28ee1c903b00e9e07fe0e8c216ef980395e2fee4430 WatchSource:0}: Error finding container b6a49c6bd26c495f2282a28ee1c903b00e9e07fe0e8c216ef980395e2fee4430: Status 404 returned error can't find the container with id b6a49c6bd26c495f2282a28ee1c903b00e9e07fe0e8c216ef980395e2fee4430
Jul 14 20:15:04.472060 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:04.471974    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-9sdrj" event={"ID":"cac4df9d-fe96-4dfa-87dc-7d36588dda49","Type":"ContainerStarted","Data":"b6a49c6bd26c495f2282a28ee1c903b00e9e07fe0e8c216ef980395e2fee4430"}
Jul 14 20:15:05.478155 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:05.478122    2560 generic.go:356] "Generic (PLEG): container finished" podID="cac4df9d-fe96-4dfa-87dc-7d36588dda49" containerID="81cd2770cf33888c1b4934963d4da7489d09cf275664e13bb6960d84b4aabe7c" exitCode=0
Jul 14 20:15:05.478532 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:05.478214    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-9sdrj" event={"ID":"cac4df9d-fe96-4dfa-87dc-7d36588dda49","Type":"ContainerDied","Data":"81cd2770cf33888c1b4934963d4da7489d09cf275664e13bb6960d84b4aabe7c"}
Jul 14 20:15:06.609853 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:06.609832    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-9sdrj"
Jul 14 20:15:06.709841 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:06.709805    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac4df9d-fe96-4dfa-87dc-7d36588dda49-maas-api-service-ca\") pod \"cac4df9d-fe96-4dfa-87dc-7d36588dda49\" (UID: \"cac4df9d-fe96-4dfa-87dc-7d36588dda49\") "
Jul 14 20:15:06.710148 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:06.710124    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cac4df9d-fe96-4dfa-87dc-7d36588dda49-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cac4df9d-fe96-4dfa-87dc-7d36588dda49" (UID: "cac4df9d-fe96-4dfa-87dc-7d36588dda49"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 20:15:06.810738 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:06.810708    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac4df9d-fe96-4dfa-87dc-7d36588dda49-maas-api-service-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:15:07.489252 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:07.489224    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-9sdrj"
Jul 14 20:15:07.489454 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:07.489264    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9ead5203-9sdrj" event={"ID":"cac4df9d-fe96-4dfa-87dc-7d36588dda49","Type":"ContainerDied","Data":"b6a49c6bd26c495f2282a28ee1c903b00e9e07fe0e8c216ef980395e2fee4430"}
Jul 14 20:15:07.489454 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:07.489309    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6a49c6bd26c495f2282a28ee1c903b00e9e07fe0e8c216ef980395e2fee4430"
Jul 14 20:15:14.475946 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:14.475909    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt"]
Jul 14 20:15:14.477441 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:14.477408    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt" podUID="f70c4887-f333-45fe-b5e3-641df4c815e2" containerName="maas-api" containerID="cri-o://cf090f584d4807c2d5b5a32a7df89594575edd50312df71ec7a084d7d052bc3d" gracePeriod=30
Jul 14 20:15:14.513247 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:14.513215    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-9ead5203-29734335-t42fq"]
Jul 14 20:15:14.518556 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:14.518533    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-9ead5203-29734335-t42fq"]
Jul 14 20:15:14.735022 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:14.734964    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt"
Jul 14 20:15:14.782314 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:14.782287    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70c4887-f333-45fe-b5e3-641df4c815e2-openshift-service-ca\") pod \"f70c4887-f333-45fe-b5e3-641df4c815e2\" (UID: \"f70c4887-f333-45fe-b5e3-641df4c815e2\") "
Jul 14 20:15:14.782463 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:14.782349    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f70c4887-f333-45fe-b5e3-641df4c815e2-maas-api-tls\") pod \"f70c4887-f333-45fe-b5e3-641df4c815e2\" (UID: \"f70c4887-f333-45fe-b5e3-641df4c815e2\") "
Jul 14 20:15:14.782463 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:14.782372    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sz5bq\" (UniqueName: \"kubernetes.io/projected/f70c4887-f333-45fe-b5e3-641df4c815e2-kube-api-access-sz5bq\") pod \"f70c4887-f333-45fe-b5e3-641df4c815e2\" (UID: \"f70c4887-f333-45fe-b5e3-641df4c815e2\") "
Jul 14 20:15:14.782641 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:14.782607    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70c4887-f333-45fe-b5e3-641df4c815e2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f70c4887-f333-45fe-b5e3-641df4c815e2" (UID: "f70c4887-f333-45fe-b5e3-641df4c815e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 20:15:14.784525 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:14.784504    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70c4887-f333-45fe-b5e3-641df4c815e2-kube-api-access-sz5bq" (OuterVolumeSpecName: "kube-api-access-sz5bq") pod "f70c4887-f333-45fe-b5e3-641df4c815e2" (UID: "f70c4887-f333-45fe-b5e3-641df4c815e2"). InnerVolumeSpecName "kube-api-access-sz5bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 20:15:14.784625 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:14.784530    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70c4887-f333-45fe-b5e3-641df4c815e2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f70c4887-f333-45fe-b5e3-641df4c815e2" (UID: "f70c4887-f333-45fe-b5e3-641df4c815e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 20:15:14.882947 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:14.882897    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70c4887-f333-45fe-b5e3-641df4c815e2-openshift-service-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:15:14.882947 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:14.882939    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f70c4887-f333-45fe-b5e3-641df4c815e2-maas-api-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:15:14.882947 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:14.882949    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz5bq\" (UniqueName: \"kubernetes.io/projected/f70c4887-f333-45fe-b5e3-641df4c815e2-kube-api-access-sz5bq\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:15:15.529415 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:15.529380    2560 generic.go:356] "Generic (PLEG): container finished" podID="f70c4887-f333-45fe-b5e3-641df4c815e2" containerID="cf090f584d4807c2d5b5a32a7df89594575edd50312df71ec7a084d7d052bc3d" exitCode=0
Jul 14 20:15:15.529880 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:15.529440    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt"
Jul 14 20:15:15.529880 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:15.529460    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt" event={"ID":"f70c4887-f333-45fe-b5e3-641df4c815e2","Type":"ContainerDied","Data":"cf090f584d4807c2d5b5a32a7df89594575edd50312df71ec7a084d7d052bc3d"}
Jul 14 20:15:15.529880 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:15.529512    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt" event={"ID":"f70c4887-f333-45fe-b5e3-641df4c815e2","Type":"ContainerDied","Data":"e8e4dc3a5520c469534ef058057875121d7c23f2c20d2e233051375625870bf1"}
Jul 14 20:15:15.529880 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:15.529534    2560 scope.go:117] "RemoveContainer" containerID="cf090f584d4807c2d5b5a32a7df89594575edd50312df71ec7a084d7d052bc3d"
Jul 14 20:15:15.538543 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:15.538524    2560 scope.go:117] "RemoveContainer" containerID="cf090f584d4807c2d5b5a32a7df89594575edd50312df71ec7a084d7d052bc3d"
Jul 14 20:15:15.538783 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:15:15.538767    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf090f584d4807c2d5b5a32a7df89594575edd50312df71ec7a084d7d052bc3d\": container with ID starting with cf090f584d4807c2d5b5a32a7df89594575edd50312df71ec7a084d7d052bc3d not found: ID does not exist" containerID="cf090f584d4807c2d5b5a32a7df89594575edd50312df71ec7a084d7d052bc3d"
Jul 14 20:15:15.538832 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:15.538791    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf090f584d4807c2d5b5a32a7df89594575edd50312df71ec7a084d7d052bc3d"} err="failed to get container status \"cf090f584d4807c2d5b5a32a7df89594575edd50312df71ec7a084d7d052bc3d\": rpc error: code = NotFound desc = could not find container \"cf090f584d4807c2d5b5a32a7df89594575edd50312df71ec7a084d7d052bc3d\": container with ID starting with cf090f584d4807c2d5b5a32a7df89594575edd50312df71ec7a084d7d052bc3d not found: ID does not exist"
Jul 14 20:15:15.554670 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:15.554646    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt"]
Jul 14 20:15:15.559063 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:15.559038    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9ead5203-6bc8df499-w6qmt"]
Jul 14 20:15:15.889710 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:15.889677    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e99f1449-dd77-461d-98b5-58834ee50b06" path="/var/lib/kubelet/pods/e99f1449-dd77-461d-98b5-58834ee50b06/volumes"
Jul 14 20:15:15.890030 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:15.890018    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f70c4887-f333-45fe-b5e3-641df4c815e2" path="/var/lib/kubelet/pods/f70c4887-f333-45fe-b5e3-641df4c815e2/volumes"
Jul 14 20:15:21.961102 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:21.961061    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d"]
Jul 14 20:15:21.961647 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:21.961627    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70c4887-f333-45fe-b5e3-641df4c815e2" containerName="maas-api"
Jul 14 20:15:21.961730 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:21.961650    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70c4887-f333-45fe-b5e3-641df4c815e2" containerName="maas-api"
Jul 14 20:15:21.961730 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:21.961696    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cac4df9d-fe96-4dfa-87dc-7d36588dda49" containerName="revoke-keys"
Jul 14 20:15:21.961730 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:21.961705    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="cac4df9d-fe96-4dfa-87dc-7d36588dda49" containerName="revoke-keys"
Jul 14 20:15:21.961872 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:21.961796    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="cac4df9d-fe96-4dfa-87dc-7d36588dda49" containerName="revoke-keys"
Jul 14 20:15:21.961872 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:21.961808    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f70c4887-f333-45fe-b5e3-641df4c815e2" containerName="maas-api"
Jul 14 20:15:21.966334 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:21.966313    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d"
Jul 14 20:15:21.968985 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:21.968951    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-96fd5154-serving-cert\""
Jul 14 20:15:21.969098 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:21.969078    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gfkms\""
Jul 14 20:15:21.979548 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:21.979525    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d"]
Jul 14 20:15:22.048424 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:22.048388    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d96b4687-423f-4b57-aa41-60de9a3a9208-maas-api-tls\") pod \"maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d\" (UID: \"d96b4687-423f-4b57-aa41-60de9a3a9208\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d"
Jul 14 20:15:22.048600 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:22.048427    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d96b4687-423f-4b57-aa41-60de9a3a9208-openshift-service-ca\") pod \"maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d\" (UID: \"d96b4687-423f-4b57-aa41-60de9a3a9208\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d"
Jul 14 20:15:22.048600 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:22.048461    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr9n4\" (UniqueName: \"kubernetes.io/projected/d96b4687-423f-4b57-aa41-60de9a3a9208-kube-api-access-vr9n4\") pod \"maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d\" (UID: \"d96b4687-423f-4b57-aa41-60de9a3a9208\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d"
Jul 14 20:15:22.149996 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:22.149958    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d96b4687-423f-4b57-aa41-60de9a3a9208-maas-api-tls\") pod \"maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d\" (UID: \"d96b4687-423f-4b57-aa41-60de9a3a9208\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d"
Jul 14 20:15:22.149996 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:22.149999    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d96b4687-423f-4b57-aa41-60de9a3a9208-openshift-service-ca\") pod \"maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d\" (UID: \"d96b4687-423f-4b57-aa41-60de9a3a9208\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d"
Jul 14 20:15:22.150224 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:22.150030    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr9n4\" (UniqueName: \"kubernetes.io/projected/d96b4687-423f-4b57-aa41-60de9a3a9208-kube-api-access-vr9n4\") pod \"maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d\" (UID: \"d96b4687-423f-4b57-aa41-60de9a3a9208\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d"
Jul 14 20:15:22.150664 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:22.150644    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d96b4687-423f-4b57-aa41-60de9a3a9208-openshift-service-ca\") pod \"maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d\" (UID: \"d96b4687-423f-4b57-aa41-60de9a3a9208\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d"
Jul 14 20:15:22.152762 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:22.152741    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d96b4687-423f-4b57-aa41-60de9a3a9208-maas-api-tls\") pod \"maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d\" (UID: \"d96b4687-423f-4b57-aa41-60de9a3a9208\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d"
Jul 14 20:15:22.160079 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:22.160058    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr9n4\" (UniqueName: \"kubernetes.io/projected/d96b4687-423f-4b57-aa41-60de9a3a9208-kube-api-access-vr9n4\") pod \"maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d\" (UID: \"d96b4687-423f-4b57-aa41-60de9a3a9208\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d"
Jul 14 20:15:22.278489 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:22.278394    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d"
Jul 14 20:15:22.416083 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:22.416047    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d"]
Jul 14 20:15:22.419775 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:15:22.419751    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd96b4687_423f_4b57_aa41_60de9a3a9208.slice/crio-86c90348bcf305fb747657f974a7504d94d7a9b77d78c1bef16b81837b1dba84 WatchSource:0}: Error finding container 86c90348bcf305fb747657f974a7504d94d7a9b77d78c1bef16b81837b1dba84: Status 404 returned error can't find the container with id 86c90348bcf305fb747657f974a7504d94d7a9b77d78c1bef16b81837b1dba84
Jul 14 20:15:22.565873 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:22.565836    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d" event={"ID":"d96b4687-423f-4b57-aa41-60de9a3a9208","Type":"ContainerStarted","Data":"86c90348bcf305fb747657f974a7504d94d7a9b77d78c1bef16b81837b1dba84"}
Jul 14 20:15:23.572056 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:23.572016    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d" event={"ID":"d96b4687-423f-4b57-aa41-60de9a3a9208","Type":"ContainerStarted","Data":"9671fd73449f15043c0ad4e10284c4615814baf537fc112f501865de205b14b7"}
Jul 14 20:15:23.572477 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:23.572092    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d"
Jul 14 20:15:23.590484 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:23.590436    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d" podStartSLOduration=1.9833114269999998 podStartE2EDuration="2.59042061s" totalImagesPullingTime="607.109183ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:15:23.588944964 +0000 UTC m=+1676.189248670" watchObservedRunningTime="2026-07-14 20:15:23.59042061 +0000 UTC m=+1676.190724313"
Jul 14 20:15:26.927013 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:26.926976    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-r2b49"]
Jul 14 20:15:26.930235 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:26.930219    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-r2b49"
Jul 14 20:15:26.932968 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:26.932942    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-56w6s\""
Jul 14 20:15:26.937575 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:26.937552    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-r2b49"]
Jul 14 20:15:26.993767 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:26.993731    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/302b4d8c-7a9f-42f8-b843-fa992daf9c6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-96fd5154-r2b49\" (UID: \"302b4d8c-7a9f-42f8-b843-fa992daf9c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-r2b49"
Jul 14 20:15:27.095151 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:27.095117    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/302b4d8c-7a9f-42f8-b843-fa992daf9c6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-96fd5154-r2b49\" (UID: \"302b4d8c-7a9f-42f8-b843-fa992daf9c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-r2b49"
Jul 14 20:15:27.095767 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:27.095747    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/302b4d8c-7a9f-42f8-b843-fa992daf9c6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-96fd5154-r2b49\" (UID: \"302b4d8c-7a9f-42f8-b843-fa992daf9c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-r2b49"
Jul 14 20:15:27.243119 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:27.243032    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-r2b49"
Jul 14 20:15:27.389348 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:27.389320    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-r2b49"]
Jul 14 20:15:27.391922 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:15:27.391891    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod302b4d8c_7a9f_42f8_b843_fa992daf9c6d.slice/crio-756c989313de4dc53c7ea1d75a3c6e7ef0cb936653d2fa95bca5edbe1f28d736 WatchSource:0}: Error finding container 756c989313de4dc53c7ea1d75a3c6e7ef0cb936653d2fa95bca5edbe1f28d736: Status 404 returned error can't find the container with id 756c989313de4dc53c7ea1d75a3c6e7ef0cb936653d2fa95bca5edbe1f28d736
Jul 14 20:15:27.593932 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:27.593899    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-r2b49" event={"ID":"302b4d8c-7a9f-42f8-b843-fa992daf9c6d","Type":"ContainerStarted","Data":"756c989313de4dc53c7ea1d75a3c6e7ef0cb936653d2fa95bca5edbe1f28d736"}
Jul 14 20:15:28.600577 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:28.600542    2560 generic.go:356] "Generic (PLEG): container finished" podID="302b4d8c-7a9f-42f8-b843-fa992daf9c6d" containerID="775bdc06e05811b95243dfb7eb8943cc5bc2586b24e888e2efda0101f24cc7bb" exitCode=7
Jul 14 20:15:28.600949 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:28.600574    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-r2b49" event={"ID":"302b4d8c-7a9f-42f8-b843-fa992daf9c6d","Type":"ContainerDied","Data":"775bdc06e05811b95243dfb7eb8943cc5bc2586b24e888e2efda0101f24cc7bb"}
Jul 14 20:15:28.600949 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:28.600852    2560 scope.go:117] "RemoveContainer" containerID="775bdc06e05811b95243dfb7eb8943cc5bc2586b24e888e2efda0101f24cc7bb"
Jul 14 20:15:29.582899 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:29.582874    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d"
Jul 14 20:15:29.608646 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:29.608551    2560 generic.go:356] "Generic (PLEG): container finished" podID="302b4d8c-7a9f-42f8-b843-fa992daf9c6d" containerID="65bc24e5a0771f50661c0ae5f4ba66b66e56c1528c7b2505b0424a20f7dcb37c" exitCode=7
Jul 14 20:15:29.608646 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:29.608585    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-r2b49" event={"ID":"302b4d8c-7a9f-42f8-b843-fa992daf9c6d","Type":"ContainerDied","Data":"65bc24e5a0771f50661c0ae5f4ba66b66e56c1528c7b2505b0424a20f7dcb37c"}
Jul 14 20:15:29.609311 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:29.608658    2560 scope.go:117] "RemoveContainer" containerID="775bdc06e05811b95243dfb7eb8943cc5bc2586b24e888e2efda0101f24cc7bb"
Jul 14 20:15:29.609311 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:29.608896    2560 scope.go:117] "RemoveContainer" containerID="65bc24e5a0771f50661c0ae5f4ba66b66e56c1528c7b2505b0424a20f7dcb37c"
Jul 14 20:15:29.609311 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:15:29.609136    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-96fd5154-r2b49_odh-ai-gateway-infra(302b4d8c-7a9f-42f8-b843-fa992daf9c6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-r2b49" podUID="302b4d8c-7a9f-42f8-b843-fa992daf9c6d"
Jul 14 20:15:30.615482 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:30.615450    2560 scope.go:117] "RemoveContainer" containerID="65bc24e5a0771f50661c0ae5f4ba66b66e56c1528c7b2505b0424a20f7dcb37c"
Jul 14 20:15:30.615913 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:15:30.615698    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-96fd5154-r2b49_odh-ai-gateway-infra(302b4d8c-7a9f-42f8-b843-fa992daf9c6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-r2b49" podUID="302b4d8c-7a9f-42f8-b843-fa992daf9c6d"
Jul 14 20:15:42.884632 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:42.884598    2560 scope.go:117] "RemoveContainer" containerID="65bc24e5a0771f50661c0ae5f4ba66b66e56c1528c7b2505b0424a20f7dcb37c"
Jul 14 20:15:43.686471 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:43.686434    2560 generic.go:356] "Generic (PLEG): container finished" podID="302b4d8c-7a9f-42f8-b843-fa992daf9c6d" containerID="f8c6fc32a7b99e0bad4f4c90433b1fa61776d4706a563988e039d1702b232b04" exitCode=0
Jul 14 20:15:43.686654 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:43.686495    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-r2b49" event={"ID":"302b4d8c-7a9f-42f8-b843-fa992daf9c6d","Type":"ContainerDied","Data":"f8c6fc32a7b99e0bad4f4c90433b1fa61776d4706a563988e039d1702b232b04"}
Jul 14 20:15:43.686654 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:43.686571    2560 scope.go:117] "RemoveContainer" containerID="65bc24e5a0771f50661c0ae5f4ba66b66e56c1528c7b2505b0424a20f7dcb37c"
Jul 14 20:15:44.717670 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:44.717637    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-r2b49"]
Jul 14 20:15:44.819950 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:44.819924    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-r2b49"
Jul 14 20:15:44.965749 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:44.965639    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/302b4d8c-7a9f-42f8-b843-fa992daf9c6d-maas-api-service-ca\") pod \"302b4d8c-7a9f-42f8-b843-fa992daf9c6d\" (UID: \"302b4d8c-7a9f-42f8-b843-fa992daf9c6d\") "
Jul 14 20:15:44.965984 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:44.965959    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/302b4d8c-7a9f-42f8-b843-fa992daf9c6d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "302b4d8c-7a9f-42f8-b843-fa992daf9c6d" (UID: "302b4d8c-7a9f-42f8-b843-fa992daf9c6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 20:15:45.067147 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:45.067108    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/302b4d8c-7a9f-42f8-b843-fa992daf9c6d-maas-api-service-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:15:45.699578 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:45.699544    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-r2b49" event={"ID":"302b4d8c-7a9f-42f8-b843-fa992daf9c6d","Type":"ContainerDied","Data":"756c989313de4dc53c7ea1d75a3c6e7ef0cb936653d2fa95bca5edbe1f28d736"}
Jul 14 20:15:45.699725 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:45.699623    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="756c989313de4dc53c7ea1d75a3c6e7ef0cb936653d2fa95bca5edbe1f28d736"
Jul 14 20:15:45.699725 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:45.699557    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-r2b49"
Jul 14 20:15:45.731460 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:45.731433    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-r2b49"]
Jul 14 20:15:45.735525 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:45.735502    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-r2b49"]
Jul 14 20:15:45.889440 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:15:45.889409    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="302b4d8c-7a9f-42f8-b843-fa992daf9c6d" path="/var/lib/kubelet/pods/302b4d8c-7a9f-42f8-b843-fa992daf9c6d/volumes"
Jul 14 20:16:16.967229 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:16.967163    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-png4z"]
Jul 14 20:16:16.967846 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:16.967584    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="302b4d8c-7a9f-42f8-b843-fa992daf9c6d" containerName="revoke-keys"
Jul 14 20:16:16.967846 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:16.967600    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="302b4d8c-7a9f-42f8-b843-fa992daf9c6d" containerName="revoke-keys"
Jul 14 20:16:16.967846 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:16.967625    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="302b4d8c-7a9f-42f8-b843-fa992daf9c6d" containerName="revoke-keys"
Jul 14 20:16:16.967846 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:16.967631    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="302b4d8c-7a9f-42f8-b843-fa992daf9c6d" containerName="revoke-keys"
Jul 14 20:16:16.967846 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:16.967691    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="302b4d8c-7a9f-42f8-b843-fa992daf9c6d" containerName="revoke-keys"
Jul 14 20:16:16.967846 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:16.967699    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="302b4d8c-7a9f-42f8-b843-fa992daf9c6d" containerName="revoke-keys"
Jul 14 20:16:16.970959 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:16.970937    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-png4z"
Jul 14 20:16:16.973284 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:16.973261    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-56w6s\""
Jul 14 20:16:16.977058 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:16.977035    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-png4z"]
Jul 14 20:16:17.055841 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:17.055802    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc362cbd-667e-4ea4-b40b-7d46034c607a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-96fd5154-png4z\" (UID: \"fc362cbd-667e-4ea4-b40b-7d46034c607a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-png4z"
Jul 14 20:16:17.156366 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:17.156318    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc362cbd-667e-4ea4-b40b-7d46034c607a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-96fd5154-png4z\" (UID: \"fc362cbd-667e-4ea4-b40b-7d46034c607a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-png4z"
Jul 14 20:16:17.156989 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:17.156971    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc362cbd-667e-4ea4-b40b-7d46034c607a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-96fd5154-png4z\" (UID: \"fc362cbd-667e-4ea4-b40b-7d46034c607a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-png4z"
Jul 14 20:16:17.283586 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:17.283547    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-png4z"
Jul 14 20:16:17.426828 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:17.426798    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-png4z"]
Jul 14 20:16:17.428474 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:16:17.428445    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc362cbd_667e_4ea4_b40b_7d46034c607a.slice/crio-fd9432435c2f5cfdfe615d5e97e198af6757552b30e81948c6615b21e1d6629a WatchSource:0}: Error finding container fd9432435c2f5cfdfe615d5e97e198af6757552b30e81948c6615b21e1d6629a: Status 404 returned error can't find the container with id fd9432435c2f5cfdfe615d5e97e198af6757552b30e81948c6615b21e1d6629a
Jul 14 20:16:17.864931 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:17.864838    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-png4z" event={"ID":"fc362cbd-667e-4ea4-b40b-7d46034c607a","Type":"ContainerStarted","Data":"17947bb3a0659e9d52acf87a592b5cfbf482ef67d319a84a2a77c69a6f06cfec"}
Jul 14 20:16:17.864931 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:17.864899    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-png4z" event={"ID":"fc362cbd-667e-4ea4-b40b-7d46034c607a","Type":"ContainerStarted","Data":"fd9432435c2f5cfdfe615d5e97e198af6757552b30e81948c6615b21e1d6629a"}
Jul 14 20:16:17.880498 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:17.880446    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-png4z" podStartSLOduration=1.880431728 podStartE2EDuration="1.880431728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:16:17.879587508 +0000 UTC m=+1730.479891210" watchObservedRunningTime="2026-07-14 20:16:17.880431728 +0000 UTC m=+1730.480735430"
Jul 14 20:16:18.878648 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:18.878617    2560 generic.go:356] "Generic (PLEG): container finished" podID="fc362cbd-667e-4ea4-b40b-7d46034c607a" containerID="17947bb3a0659e9d52acf87a592b5cfbf482ef67d319a84a2a77c69a6f06cfec" exitCode=0
Jul 14 20:16:18.879116 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:18.878709    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-png4z" event={"ID":"fc362cbd-667e-4ea4-b40b-7d46034c607a","Type":"ContainerDied","Data":"17947bb3a0659e9d52acf87a592b5cfbf482ef67d319a84a2a77c69a6f06cfec"}
Jul 14 20:16:20.015413 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:20.015389    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-png4z"
Jul 14 20:16:20.086741 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:20.086701    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc362cbd-667e-4ea4-b40b-7d46034c607a-maas-api-service-ca\") pod \"fc362cbd-667e-4ea4-b40b-7d46034c607a\" (UID: \"fc362cbd-667e-4ea4-b40b-7d46034c607a\") "
Jul 14 20:16:20.087110 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:20.087083    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc362cbd-667e-4ea4-b40b-7d46034c607a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fc362cbd-667e-4ea4-b40b-7d46034c607a" (UID: "fc362cbd-667e-4ea4-b40b-7d46034c607a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 20:16:20.187571 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:20.187482    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc362cbd-667e-4ea4-b40b-7d46034c607a-maas-api-service-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:16:20.889623 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:20.889584    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-png4z"
Jul 14 20:16:20.889794 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:20.889620    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96fd5154-png4z" event={"ID":"fc362cbd-667e-4ea4-b40b-7d46034c607a","Type":"ContainerDied","Data":"fd9432435c2f5cfdfe615d5e97e198af6757552b30e81948c6615b21e1d6629a"}
Jul 14 20:16:20.889794 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:20.889664    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd9432435c2f5cfdfe615d5e97e198af6757552b30e81948c6615b21e1d6629a"
Jul 14 20:16:27.460756 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:27.460721    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d"]
Jul 14 20:16:27.461518 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:27.461462    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d" podUID="d96b4687-423f-4b57-aa41-60de9a3a9208" containerName="maas-api" containerID="cri-o://9671fd73449f15043c0ad4e10284c4615814baf537fc112f501865de205b14b7" gracePeriod=30
Jul 14 20:16:27.709391 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:27.709370    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d"
Jul 14 20:16:27.858858 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:27.858821    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vr9n4\" (UniqueName: \"kubernetes.io/projected/d96b4687-423f-4b57-aa41-60de9a3a9208-kube-api-access-vr9n4\") pod \"d96b4687-423f-4b57-aa41-60de9a3a9208\" (UID: \"d96b4687-423f-4b57-aa41-60de9a3a9208\") "
Jul 14 20:16:27.859067 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:27.858872    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d96b4687-423f-4b57-aa41-60de9a3a9208-openshift-service-ca\") pod \"d96b4687-423f-4b57-aa41-60de9a3a9208\" (UID: \"d96b4687-423f-4b57-aa41-60de9a3a9208\") "
Jul 14 20:16:27.859067 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:27.858981    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d96b4687-423f-4b57-aa41-60de9a3a9208-maas-api-tls\") pod \"d96b4687-423f-4b57-aa41-60de9a3a9208\" (UID: \"d96b4687-423f-4b57-aa41-60de9a3a9208\") "
Jul 14 20:16:27.859339 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:27.859313    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d96b4687-423f-4b57-aa41-60de9a3a9208-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d96b4687-423f-4b57-aa41-60de9a3a9208" (UID: "d96b4687-423f-4b57-aa41-60de9a3a9208"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 20:16:27.861164 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:27.861141    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d96b4687-423f-4b57-aa41-60de9a3a9208-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d96b4687-423f-4b57-aa41-60de9a3a9208" (UID: "d96b4687-423f-4b57-aa41-60de9a3a9208"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 20:16:27.861247 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:27.861203    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d96b4687-423f-4b57-aa41-60de9a3a9208-kube-api-access-vr9n4" (OuterVolumeSpecName: "kube-api-access-vr9n4") pod "d96b4687-423f-4b57-aa41-60de9a3a9208" (UID: "d96b4687-423f-4b57-aa41-60de9a3a9208"). InnerVolumeSpecName "kube-api-access-vr9n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 20:16:27.924557 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:27.924521    2560 generic.go:356] "Generic (PLEG): container finished" podID="d96b4687-423f-4b57-aa41-60de9a3a9208" containerID="9671fd73449f15043c0ad4e10284c4615814baf537fc112f501865de205b14b7" exitCode=0
Jul 14 20:16:27.924704 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:27.924568    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d"
Jul 14 20:16:27.924704 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:27.924603    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d" event={"ID":"d96b4687-423f-4b57-aa41-60de9a3a9208","Type":"ContainerDied","Data":"9671fd73449f15043c0ad4e10284c4615814baf537fc112f501865de205b14b7"}
Jul 14 20:16:27.924704 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:27.924658    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d" event={"ID":"d96b4687-423f-4b57-aa41-60de9a3a9208","Type":"ContainerDied","Data":"86c90348bcf305fb747657f974a7504d94d7a9b77d78c1bef16b81837b1dba84"}
Jul 14 20:16:27.924704 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:27.924686    2560 scope.go:117] "RemoveContainer" containerID="9671fd73449f15043c0ad4e10284c4615814baf537fc112f501865de205b14b7"
Jul 14 20:16:27.933966 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:27.933943    2560 scope.go:117] "RemoveContainer" containerID="9671fd73449f15043c0ad4e10284c4615814baf537fc112f501865de205b14b7"
Jul 14 20:16:27.934288 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:16:27.934262    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9671fd73449f15043c0ad4e10284c4615814baf537fc112f501865de205b14b7\": container with ID starting with 9671fd73449f15043c0ad4e10284c4615814baf537fc112f501865de205b14b7 not found: ID does not exist" containerID="9671fd73449f15043c0ad4e10284c4615814baf537fc112f501865de205b14b7"
Jul 14 20:16:27.934347 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:27.934299    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9671fd73449f15043c0ad4e10284c4615814baf537fc112f501865de205b14b7"} err="failed to get container status \"9671fd73449f15043c0ad4e10284c4615814baf537fc112f501865de205b14b7\": rpc error: code = NotFound desc = could not find container \"9671fd73449f15043c0ad4e10284c4615814baf537fc112f501865de205b14b7\": container with ID starting with 9671fd73449f15043c0ad4e10284c4615814baf537fc112f501865de205b14b7 not found: ID does not exist"
Jul 14 20:16:27.942243 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:27.942217    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d"]
Jul 14 20:16:27.946245 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:27.946222    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-96fd5154-5b69664db5-tzs2d"]
Jul 14 20:16:27.960590 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:27.960566    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d96b4687-423f-4b57-aa41-60de9a3a9208-maas-api-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:16:27.960590 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:27.960590    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr9n4\" (UniqueName: \"kubernetes.io/projected/d96b4687-423f-4b57-aa41-60de9a3a9208-kube-api-access-vr9n4\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:16:27.960732 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:27.960604    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d96b4687-423f-4b57-aa41-60de9a3a9208-openshift-service-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:16:29.890207 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:29.890152    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d96b4687-423f-4b57-aa41-60de9a3a9208" path="/var/lib/kubelet/pods/d96b4687-423f-4b57-aa41-60de9a3a9208/volumes"
Jul 14 20:16:44.641159 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:44.641126    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2"]
Jul 14 20:16:44.641698 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:44.641604    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="302b4d8c-7a9f-42f8-b843-fa992daf9c6d" containerName="revoke-keys"
Jul 14 20:16:44.641698 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:44.641622    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="302b4d8c-7a9f-42f8-b843-fa992daf9c6d" containerName="revoke-keys"
Jul 14 20:16:44.641698 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:44.641646    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc362cbd-667e-4ea4-b40b-7d46034c607a" containerName="revoke-keys"
Jul 14 20:16:44.641698 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:44.641652    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc362cbd-667e-4ea4-b40b-7d46034c607a" containerName="revoke-keys"
Jul 14 20:16:44.641698 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:44.641659    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d96b4687-423f-4b57-aa41-60de9a3a9208" containerName="maas-api"
Jul 14 20:16:44.641698 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:44.641665    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d96b4687-423f-4b57-aa41-60de9a3a9208" containerName="maas-api"
Jul 14 20:16:44.642039 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:44.641729    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d96b4687-423f-4b57-aa41-60de9a3a9208" containerName="maas-api"
Jul 14 20:16:44.642039 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:44.641740    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc362cbd-667e-4ea4-b40b-7d46034c607a" containerName="revoke-keys"
Jul 14 20:16:44.642039 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:44.641747    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="302b4d8c-7a9f-42f8-b843-fa992daf9c6d" containerName="revoke-keys"
Jul 14 20:16:44.646213 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:44.646193    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2"
Jul 14 20:16:44.648686 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:44.648655    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-f02785bb-serving-cert\""
Jul 14 20:16:44.648923 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:44.648737    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gfkms\""
Jul 14 20:16:44.652965 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:44.652939    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2"]
Jul 14 20:16:44.704343 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:44.704308    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac-openshift-service-ca\") pod \"maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2\" (UID: \"ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2"
Jul 14 20:16:44.704525 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:44.704367    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqj5b\" (UniqueName: \"kubernetes.io/projected/ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac-kube-api-access-lqj5b\") pod \"maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2\" (UID: \"ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2"
Jul 14 20:16:44.704525 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:44.704474    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac-maas-api-tls\") pod \"maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2\" (UID: \"ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2"
Jul 14 20:16:44.805149 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:44.805116    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac-maas-api-tls\") pod \"maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2\" (UID: \"ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2"
Jul 14 20:16:44.805149 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:44.805154    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac-openshift-service-ca\") pod \"maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2\" (UID: \"ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2"
Jul 14 20:16:44.805390 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:44.805219    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqj5b\" (UniqueName: \"kubernetes.io/projected/ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac-kube-api-access-lqj5b\") pod \"maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2\" (UID: \"ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2"
Jul 14 20:16:44.805863 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:44.805845    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac-openshift-service-ca\") pod \"maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2\" (UID: \"ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2"
Jul 14 20:16:44.807674 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:44.807654    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac-maas-api-tls\") pod \"maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2\" (UID: \"ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2"
Jul 14 20:16:44.813525 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:44.813500    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqj5b\" (UniqueName: \"kubernetes.io/projected/ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac-kube-api-access-lqj5b\") pod \"maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2\" (UID: \"ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2"
Jul 14 20:16:44.957803 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:44.957709    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2"
Jul 14 20:16:45.101969 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:45.101923    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2"]
Jul 14 20:16:45.104938 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:16:45.104908    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecfd2a36_4b36_45fa_b3fe_96e1fe9ed5ac.slice/crio-fc3a092799bc371652654c58c4ea88c271426553150e21f32f7133f8dfd469ef WatchSource:0}: Error finding container fc3a092799bc371652654c58c4ea88c271426553150e21f32f7133f8dfd469ef: Status 404 returned error can't find the container with id fc3a092799bc371652654c58c4ea88c271426553150e21f32f7133f8dfd469ef
Jul 14 20:16:46.021869 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:46.021833    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2" event={"ID":"ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac","Type":"ContainerStarted","Data":"fc3a092799bc371652654c58c4ea88c271426553150e21f32f7133f8dfd469ef"}
Jul 14 20:16:47.028729 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:47.028643    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2" event={"ID":"ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac","Type":"ContainerStarted","Data":"b49b520910a698f12c7dce3af3967dabcaaf3b4834c55e74ca63f2405f5a9a06"}
Jul 14 20:16:47.029122 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:47.028769    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2"
Jul 14 20:16:47.047637 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:47.047592    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2" podStartSLOduration=1.410323834 podStartE2EDuration="3.04757352s" totalImagesPullingTime="1.637249686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:16:47.046162855 +0000 UTC m=+1759.646466569" watchObservedRunningTime="2026-07-14 20:16:47.04757352 +0000 UTC m=+1759.647877224"
Jul 14 20:16:48.981797 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:48.981758    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g"]
Jul 14 20:16:48.985472 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:48.985450    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g"
Jul 14 20:16:48.987805 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:48.987782    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-56w6s\""
Jul 14 20:16:48.992553 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:48.992530    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g"]
Jul 14 20:16:49.044366 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:49.044332    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f6696d5-a302-401a-985d-1ae61ecd0a8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g\" (UID: \"9f6696d5-a302-401a-985d-1ae61ecd0a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g"
Jul 14 20:16:49.145451 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:49.145418    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f6696d5-a302-401a-985d-1ae61ecd0a8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g\" (UID: \"9f6696d5-a302-401a-985d-1ae61ecd0a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g"
Jul 14 20:16:49.146030 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:49.146008    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f6696d5-a302-401a-985d-1ae61ecd0a8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g\" (UID: \"9f6696d5-a302-401a-985d-1ae61ecd0a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g"
Jul 14 20:16:49.299284 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:49.299251    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g"
Jul 14 20:16:49.441854 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:49.441829    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g"]
Jul 14 20:16:49.445045 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:16:49.445014    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f6696d5_a302_401a_985d_1ae61ecd0a8b.slice/crio-b907641055a46096ea76c4ff60c39e5320d8a99a9f3b3760636974f9a8fe9c16 WatchSource:0}: Error finding container b907641055a46096ea76c4ff60c39e5320d8a99a9f3b3760636974f9a8fe9c16: Status 404 returned error can't find the container with id b907641055a46096ea76c4ff60c39e5320d8a99a9f3b3760636974f9a8fe9c16
Jul 14 20:16:50.045902 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:50.045866    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g" event={"ID":"9f6696d5-a302-401a-985d-1ae61ecd0a8b","Type":"ContainerStarted","Data":"1805a436c1e0d2070b2e099de32956ca5d1caf6b7374d9dbdc9964deeded38fb"}
Jul 14 20:16:50.046321 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:50.045924    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g" event={"ID":"9f6696d5-a302-401a-985d-1ae61ecd0a8b","Type":"ContainerStarted","Data":"b907641055a46096ea76c4ff60c39e5320d8a99a9f3b3760636974f9a8fe9c16"}
Jul 14 20:16:50.062841 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:50.062791    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g" podStartSLOduration=2.062775859 podStartE2EDuration="2.062775859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:16:50.060803014 +0000 UTC m=+1762.661106706" watchObservedRunningTime="2026-07-14 20:16:50.062775859 +0000 UTC m=+1762.663079583"
Jul 14 20:16:51.052055 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:51.052023    2560 generic.go:356] "Generic (PLEG): container finished" podID="9f6696d5-a302-401a-985d-1ae61ecd0a8b" containerID="1805a436c1e0d2070b2e099de32956ca5d1caf6b7374d9dbdc9964deeded38fb" exitCode=7
Jul 14 20:16:51.052501 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:51.052112    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g" event={"ID":"9f6696d5-a302-401a-985d-1ae61ecd0a8b","Type":"ContainerDied","Data":"1805a436c1e0d2070b2e099de32956ca5d1caf6b7374d9dbdc9964deeded38fb"}
Jul 14 20:16:51.052501 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:51.052349    2560 scope.go:117] "RemoveContainer" containerID="1805a436c1e0d2070b2e099de32956ca5d1caf6b7374d9dbdc9964deeded38fb"
Jul 14 20:16:52.059063 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:52.059030    2560 generic.go:356] "Generic (PLEG): container finished" podID="9f6696d5-a302-401a-985d-1ae61ecd0a8b" containerID="71c550d1f6fdb2c527f3257444dd0aa5e03052a80bab55930392799c41b25c2c" exitCode=7
Jul 14 20:16:52.059486 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:52.059121    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g" event={"ID":"9f6696d5-a302-401a-985d-1ae61ecd0a8b","Type":"ContainerDied","Data":"71c550d1f6fdb2c527f3257444dd0aa5e03052a80bab55930392799c41b25c2c"}
Jul 14 20:16:52.059486 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:52.059198    2560 scope.go:117] "RemoveContainer" containerID="1805a436c1e0d2070b2e099de32956ca5d1caf6b7374d9dbdc9964deeded38fb"
Jul 14 20:16:52.059486 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:52.059360    2560 scope.go:117] "RemoveContainer" containerID="71c550d1f6fdb2c527f3257444dd0aa5e03052a80bab55930392799c41b25c2c"
Jul 14 20:16:52.059650 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:16:52.059571    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g_odh-ai-gateway-infra(9f6696d5-a302-401a-985d-1ae61ecd0a8b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g" podUID="9f6696d5-a302-401a-985d-1ae61ecd0a8b"
Jul 14 20:16:53.038735 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:53.038704    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2"
Jul 14 20:16:53.066557 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:16:53.066527    2560 scope.go:117] "RemoveContainer" containerID="71c550d1f6fdb2c527f3257444dd0aa5e03052a80bab55930392799c41b25c2c"
Jul 14 20:16:53.066942 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:16:53.066742    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g_odh-ai-gateway-infra(9f6696d5-a302-401a-985d-1ae61ecd0a8b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g" podUID="9f6696d5-a302-401a-985d-1ae61ecd0a8b"
Jul 14 20:17:05.884051 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:05.884016    2560 scope.go:117] "RemoveContainer" containerID="71c550d1f6fdb2c527f3257444dd0aa5e03052a80bab55930392799c41b25c2c"
Jul 14 20:17:06.136548 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:06.136485    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g" event={"ID":"9f6696d5-a302-401a-985d-1ae61ecd0a8b","Type":"ContainerStarted","Data":"687896a4315cba84db072b41f903a7c937ba55ca1296cf3a5d4bfd8c81de59a7"}
Jul 14 20:17:07.143299 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:07.143269    2560 generic.go:356] "Generic (PLEG): container finished" podID="9f6696d5-a302-401a-985d-1ae61ecd0a8b" containerID="687896a4315cba84db072b41f903a7c937ba55ca1296cf3a5d4bfd8c81de59a7" exitCode=0
Jul 14 20:17:07.143687 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:07.143344    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g" event={"ID":"9f6696d5-a302-401a-985d-1ae61ecd0a8b","Type":"ContainerDied","Data":"687896a4315cba84db072b41f903a7c937ba55ca1296cf3a5d4bfd8c81de59a7"}
Jul 14 20:17:07.143687 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:07.143408    2560 scope.go:117] "RemoveContainer" containerID="71c550d1f6fdb2c527f3257444dd0aa5e03052a80bab55930392799c41b25c2c"
Jul 14 20:17:07.164667 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:07.164639    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g"]
Jul 14 20:17:08.284649 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:08.284621    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g"
Jul 14 20:17:08.417514 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:08.417418    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f6696d5-a302-401a-985d-1ae61ecd0a8b-maas-api-service-ca\") pod \"9f6696d5-a302-401a-985d-1ae61ecd0a8b\" (UID: \"9f6696d5-a302-401a-985d-1ae61ecd0a8b\") "
Jul 14 20:17:08.417777 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:08.417750    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f6696d5-a302-401a-985d-1ae61ecd0a8b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9f6696d5-a302-401a-985d-1ae61ecd0a8b" (UID: "9f6696d5-a302-401a-985d-1ae61ecd0a8b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 20:17:08.518907 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:08.518877    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f6696d5-a302-401a-985d-1ae61ecd0a8b-maas-api-service-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:17:09.156722 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:09.156689    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g"
Jul 14 20:17:09.156893 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:09.156721    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g" event={"ID":"9f6696d5-a302-401a-985d-1ae61ecd0a8b","Type":"ContainerDied","Data":"b907641055a46096ea76c4ff60c39e5320d8a99a9f3b3760636974f9a8fe9c16"}
Jul 14 20:17:09.156893 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:09.156787    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b907641055a46096ea76c4ff60c39e5320d8a99a9f3b3760636974f9a8fe9c16"
Jul 14 20:17:09.189065 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:09.189028    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g"]
Jul 14 20:17:09.191276 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:09.191255    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-tvc4g"]
Jul 14 20:17:09.889292 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:09.889263    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f6696d5-a302-401a-985d-1ae61ecd0a8b" path="/var/lib/kubelet/pods/9f6696d5-a302-401a-985d-1ae61ecd0a8b/volumes"
Jul 14 20:17:27.948527 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:27.948498    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tzr6_4a36af4c-9eac-465f-9a0e-d65a382530f7/ovn-acl-logging/0.log"
Jul 14 20:17:27.956843 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:27.956823    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tzr6_4a36af4c-9eac-465f-9a0e-d65a382530f7/ovn-acl-logging/0.log"
Jul 14 20:17:49.028902 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:49.028859    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-xlwsk"]
Jul 14 20:17:49.029516 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:49.029495    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f6696d5-a302-401a-985d-1ae61ecd0a8b" containerName="revoke-keys"
Jul 14 20:17:49.029564 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:49.029521    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f6696d5-a302-401a-985d-1ae61ecd0a8b" containerName="revoke-keys"
Jul 14 20:17:49.029564 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:49.029536    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f6696d5-a302-401a-985d-1ae61ecd0a8b" containerName="revoke-keys"
Jul 14 20:17:49.029564 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:49.029545    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f6696d5-a302-401a-985d-1ae61ecd0a8b" containerName="revoke-keys"
Jul 14 20:17:49.029667 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:49.029567    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f6696d5-a302-401a-985d-1ae61ecd0a8b" containerName="revoke-keys"
Jul 14 20:17:49.029667 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:49.029576    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f6696d5-a302-401a-985d-1ae61ecd0a8b" containerName="revoke-keys"
Jul 14 20:17:49.029667 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:49.029665    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f6696d5-a302-401a-985d-1ae61ecd0a8b" containerName="revoke-keys"
Jul 14 20:17:49.029766 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:49.029677    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f6696d5-a302-401a-985d-1ae61ecd0a8b" containerName="revoke-keys"
Jul 14 20:17:49.033248 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:49.033230    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-xlwsk"
Jul 14 20:17:49.036049 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:49.036017    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-56w6s\""
Jul 14 20:17:49.038501 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:49.038455    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-xlwsk"]
Jul 14 20:17:49.082857 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:49.082821    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92654054-1bfa-4df2-bf8a-e70920c1cd5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f02785bb-xlwsk\" (UID: \"92654054-1bfa-4df2-bf8a-e70920c1cd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-xlwsk"
Jul 14 20:17:49.183504 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:49.183470    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92654054-1bfa-4df2-bf8a-e70920c1cd5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f02785bb-xlwsk\" (UID: \"92654054-1bfa-4df2-bf8a-e70920c1cd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-xlwsk"
Jul 14 20:17:49.184098 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:49.184078    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92654054-1bfa-4df2-bf8a-e70920c1cd5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f02785bb-xlwsk\" (UID: \"92654054-1bfa-4df2-bf8a-e70920c1cd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-xlwsk"
Jul 14 20:17:49.347315 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:49.347282    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-xlwsk"
Jul 14 20:17:49.491947 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:49.491912    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-xlwsk"]
Jul 14 20:17:49.493560 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:17:49.493533    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92654054_1bfa_4df2_bf8a_e70920c1cd5f.slice/crio-0fde913585a8f89469cf81aae335b90f274249530993d196fb20269ac8b71331 WatchSource:0}: Error finding container 0fde913585a8f89469cf81aae335b90f274249530993d196fb20269ac8b71331: Status 404 returned error can't find the container with id 0fde913585a8f89469cf81aae335b90f274249530993d196fb20269ac8b71331
Jul 14 20:17:50.362447 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:50.362410    2560 generic.go:356] "Generic (PLEG): container finished" podID="92654054-1bfa-4df2-bf8a-e70920c1cd5f" containerID="72956b42305e4aeb24a5f8dc65fd4881b359c50c4bd1127ec77c2940a646913a" exitCode=0
Jul 14 20:17:50.362842 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:50.362504    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-xlwsk" event={"ID":"92654054-1bfa-4df2-bf8a-e70920c1cd5f","Type":"ContainerDied","Data":"72956b42305e4aeb24a5f8dc65fd4881b359c50c4bd1127ec77c2940a646913a"}
Jul 14 20:17:50.362842 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:50.362553    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-xlwsk" event={"ID":"92654054-1bfa-4df2-bf8a-e70920c1cd5f","Type":"ContainerStarted","Data":"0fde913585a8f89469cf81aae335b90f274249530993d196fb20269ac8b71331"}
Jul 14 20:17:51.498978 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:51.498954    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-xlwsk"
Jul 14 20:17:51.606957 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:51.606924    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92654054-1bfa-4df2-bf8a-e70920c1cd5f-maas-api-service-ca\") pod \"92654054-1bfa-4df2-bf8a-e70920c1cd5f\" (UID: \"92654054-1bfa-4df2-bf8a-e70920c1cd5f\") "
Jul 14 20:17:51.607311 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:51.607287    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92654054-1bfa-4df2-bf8a-e70920c1cd5f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "92654054-1bfa-4df2-bf8a-e70920c1cd5f" (UID: "92654054-1bfa-4df2-bf8a-e70920c1cd5f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 20:17:51.708093 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:51.708006    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92654054-1bfa-4df2-bf8a-e70920c1cd5f-maas-api-service-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:17:52.374435 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:52.374403    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-xlwsk"
Jul 14 20:17:52.374633 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:52.374436    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f02785bb-xlwsk" event={"ID":"92654054-1bfa-4df2-bf8a-e70920c1cd5f","Type":"ContainerDied","Data":"0fde913585a8f89469cf81aae335b90f274249530993d196fb20269ac8b71331"}
Jul 14 20:17:52.374633 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:52.374480    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fde913585a8f89469cf81aae335b90f274249530993d196fb20269ac8b71331"
Jul 14 20:17:59.558927 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:59.558884    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2"]
Jul 14 20:17:59.559484 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:59.559117    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2" podUID="ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac" containerName="maas-api" containerID="cri-o://b49b520910a698f12c7dce3af3967dabcaaf3b4834c55e74ca63f2405f5a9a06" gracePeriod=30
Jul 14 20:17:59.807435 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:59.807410    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2"
Jul 14 20:17:59.876908 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:59.876824    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac-maas-api-tls\") pod \"ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac\" (UID: \"ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac\") "
Jul 14 20:17:59.876908 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:59.876875    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac-openshift-service-ca\") pod \"ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac\" (UID: \"ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac\") "
Jul 14 20:17:59.877097 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:59.876965    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqj5b\" (UniqueName: \"kubernetes.io/projected/ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac-kube-api-access-lqj5b\") pod \"ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac\" (UID: \"ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac\") "
Jul 14 20:17:59.877346 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:59.877318    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac" (UID: "ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 20:17:59.879083 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:59.879056    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac" (UID: "ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 20:17:59.879083 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:59.879078    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac-kube-api-access-lqj5b" (OuterVolumeSpecName: "kube-api-access-lqj5b") pod "ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac" (UID: "ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac"). InnerVolumeSpecName "kube-api-access-lqj5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 20:17:59.978075 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:59.978038    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqj5b\" (UniqueName: \"kubernetes.io/projected/ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac-kube-api-access-lqj5b\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:17:59.978075 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:59.978072    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac-maas-api-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:17:59.978075 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:17:59.978083    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac-openshift-service-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:18:00.414937 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:18:00.414905    2560 generic.go:356] "Generic (PLEG): container finished" podID="ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac" containerID="b49b520910a698f12c7dce3af3967dabcaaf3b4834c55e74ca63f2405f5a9a06" exitCode=0
Jul 14 20:18:00.415112 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:18:00.414956    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2"
Jul 14 20:18:00.415112 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:18:00.414995    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2" event={"ID":"ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac","Type":"ContainerDied","Data":"b49b520910a698f12c7dce3af3967dabcaaf3b4834c55e74ca63f2405f5a9a06"}
Jul 14 20:18:00.415112 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:18:00.415051    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2" event={"ID":"ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac","Type":"ContainerDied","Data":"fc3a092799bc371652654c58c4ea88c271426553150e21f32f7133f8dfd469ef"}
Jul 14 20:18:00.415112 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:18:00.415079    2560 scope.go:117] "RemoveContainer" containerID="b49b520910a698f12c7dce3af3967dabcaaf3b4834c55e74ca63f2405f5a9a06"
Jul 14 20:18:00.424069 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:18:00.424052    2560 scope.go:117] "RemoveContainer" containerID="b49b520910a698f12c7dce3af3967dabcaaf3b4834c55e74ca63f2405f5a9a06"
Jul 14 20:18:00.424341 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:18:00.424325    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b49b520910a698f12c7dce3af3967dabcaaf3b4834c55e74ca63f2405f5a9a06\": container with ID starting with b49b520910a698f12c7dce3af3967dabcaaf3b4834c55e74ca63f2405f5a9a06 not found: ID does not exist" containerID="b49b520910a698f12c7dce3af3967dabcaaf3b4834c55e74ca63f2405f5a9a06"
Jul 14 20:18:00.424398 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:18:00.424349    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b49b520910a698f12c7dce3af3967dabcaaf3b4834c55e74ca63f2405f5a9a06"} err="failed to get container status \"b49b520910a698f12c7dce3af3967dabcaaf3b4834c55e74ca63f2405f5a9a06\": rpc error: code = NotFound desc = could not find container \"b49b520910a698f12c7dce3af3967dabcaaf3b4834c55e74ca63f2405f5a9a06\": container with ID starting with b49b520910a698f12c7dce3af3967dabcaaf3b4834c55e74ca63f2405f5a9a06 not found: ID does not exist"
Jul 14 20:18:00.437100 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:18:00.437065    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2"]
Jul 14 20:18:00.437264 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:18:00.437120    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f02785bb-766c8f54b6-6vqw2"]
Jul 14 20:18:01.889552 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:18:01.889516    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac" path="/var/lib/kubelet/pods/ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac/volumes"
Jul 14 20:19:28.042107 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:28.042077    2560 scope.go:117] "RemoveContainer" containerID="05c3bc90990df336b8c557032ae294b395ce8d40e90adce1412c662f8bea7e1d"
Jul 14 20:19:54.877736 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:54.877698    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr"]
Jul 14 20:19:54.878129 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:54.878115    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac" containerName="maas-api"
Jul 14 20:19:54.878197 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:54.878131    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac" containerName="maas-api"
Jul 14 20:19:54.878197 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:54.878146    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92654054-1bfa-4df2-bf8a-e70920c1cd5f" containerName="revoke-keys"
Jul 14 20:19:54.878197 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:54.878151    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="92654054-1bfa-4df2-bf8a-e70920c1cd5f" containerName="revoke-keys"
Jul 14 20:19:54.878306 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:54.878243    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f6696d5-a302-401a-985d-1ae61ecd0a8b" containerName="revoke-keys"
Jul 14 20:19:54.878306 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:54.878252    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecfd2a36-4b36-45fa-b3fe-96e1fe9ed5ac" containerName="maas-api"
Jul 14 20:19:54.878306 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:54.878265    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="92654054-1bfa-4df2-bf8a-e70920c1cd5f" containerName="revoke-keys"
Jul 14 20:19:54.881459 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:54.881440    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr"
Jul 14 20:19:54.884057 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:54.884037    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-407eaab7-serving-cert\""
Jul 14 20:19:54.884160 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:54.884081    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gfkms\""
Jul 14 20:19:54.889811 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:54.889752    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr"]
Jul 14 20:19:54.898100 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:54.898077    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f4dd92-8fce-48b7-a88d-17cb03152f1e-openshift-service-ca\") pod \"maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr\" (UID: \"a8f4dd92-8fce-48b7-a88d-17cb03152f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr"
Jul 14 20:19:54.898261 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:54.898204    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8f4dd92-8fce-48b7-a88d-17cb03152f1e-maas-api-tls\") pod \"maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr\" (UID: \"a8f4dd92-8fce-48b7-a88d-17cb03152f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr"
Jul 14 20:19:54.898261 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:54.898239    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqs4n\" (UniqueName: \"kubernetes.io/projected/a8f4dd92-8fce-48b7-a88d-17cb03152f1e-kube-api-access-rqs4n\") pod \"maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr\" (UID: \"a8f4dd92-8fce-48b7-a88d-17cb03152f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr"
Jul 14 20:19:54.999492 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:54.999455    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8f4dd92-8fce-48b7-a88d-17cb03152f1e-maas-api-tls\") pod \"maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr\" (UID: \"a8f4dd92-8fce-48b7-a88d-17cb03152f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr"
Jul 14 20:19:54.999667 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:54.999498    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqs4n\" (UniqueName: \"kubernetes.io/projected/a8f4dd92-8fce-48b7-a88d-17cb03152f1e-kube-api-access-rqs4n\") pod \"maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr\" (UID: \"a8f4dd92-8fce-48b7-a88d-17cb03152f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr"
Jul 14 20:19:54.999667 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:54.999592    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f4dd92-8fce-48b7-a88d-17cb03152f1e-openshift-service-ca\") pod \"maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr\" (UID: \"a8f4dd92-8fce-48b7-a88d-17cb03152f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr"
Jul 14 20:19:54.999667 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:19:54.999611    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-serving-cert: secret "maas-api-e2e-mt-407eaab7-serving-cert" not found
Jul 14 20:19:54.999777 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:19:54.999685    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8f4dd92-8fce-48b7-a88d-17cb03152f1e-maas-api-tls podName:a8f4dd92-8fce-48b7-a88d-17cb03152f1e nodeName:}" failed. No retries permitted until 2026-07-14 20:19:55.499665545 +0000 UTC m=+1948.099969227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a8f4dd92-8fce-48b7-a88d-17cb03152f1e-maas-api-tls") pod "maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr" (UID: "a8f4dd92-8fce-48b7-a88d-17cb03152f1e") : secret "maas-api-e2e-mt-407eaab7-serving-cert" not found
Jul 14 20:19:55.000253 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:55.000230    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f4dd92-8fce-48b7-a88d-17cb03152f1e-openshift-service-ca\") pod \"maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr\" (UID: \"a8f4dd92-8fce-48b7-a88d-17cb03152f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr"
Jul 14 20:19:55.012796 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:55.012765    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqs4n\" (UniqueName: \"kubernetes.io/projected/a8f4dd92-8fce-48b7-a88d-17cb03152f1e-kube-api-access-rqs4n\") pod \"maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr\" (UID: \"a8f4dd92-8fce-48b7-a88d-17cb03152f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr"
Jul 14 20:19:55.504839 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:55.504808    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8f4dd92-8fce-48b7-a88d-17cb03152f1e-maas-api-tls\") pod \"maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr\" (UID: \"a8f4dd92-8fce-48b7-a88d-17cb03152f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr"
Jul 14 20:19:55.507455 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:55.507425    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8f4dd92-8fce-48b7-a88d-17cb03152f1e-maas-api-tls\") pod \"maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr\" (UID: \"a8f4dd92-8fce-48b7-a88d-17cb03152f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr"
Jul 14 20:19:55.793280 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:55.793243    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr"
Jul 14 20:19:55.939491 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:55.939463    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr"]
Jul 14 20:19:55.941938 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:19:55.941909    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f4dd92_8fce_48b7_a88d_17cb03152f1e.slice/crio-a79773fe64b727dafbb382aaaf2fb31074db86dbd3f273f26bd6d01fad503f3c WatchSource:0}: Error finding container a79773fe64b727dafbb382aaaf2fb31074db86dbd3f273f26bd6d01fad503f3c: Status 404 returned error can't find the container with id a79773fe64b727dafbb382aaaf2fb31074db86dbd3f273f26bd6d01fad503f3c
Jul 14 20:19:55.943328 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:55.943312    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 20:19:56.005159 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:56.005114    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr" event={"ID":"a8f4dd92-8fce-48b7-a88d-17cb03152f1e","Type":"ContainerStarted","Data":"a79773fe64b727dafbb382aaaf2fb31074db86dbd3f273f26bd6d01fad503f3c"}
Jul 14 20:19:57.012153 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:57.012115    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr" event={"ID":"a8f4dd92-8fce-48b7-a88d-17cb03152f1e","Type":"ContainerStarted","Data":"9474c4316288071dbec51a0e07c4749efb88ee5f7bce78e08c47e2aa848b9d25"}
Jul 14 20:19:57.012591 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:57.012314    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr"
Jul 14 20:19:57.029078 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:19:57.029033    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr" podStartSLOduration=2.503772697 podStartE2EDuration="3.02901831s" totalImagesPullingTime="525.245613ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:19:57.027539214 +0000 UTC m=+1949.627842915" watchObservedRunningTime="2026-07-14 20:19:57.02901831 +0000 UTC m=+1949.629322027"
Jul 14 20:20:03.022182 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:03.022141    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr"
Jul 14 20:20:07.124610 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:07.124564    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-407eaab7-4tfp2"]
Jul 14 20:20:07.128303 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:07.128280    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-407eaab7-4tfp2"
Jul 14 20:20:07.130678 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:07.130658    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-56w6s\""
Jul 14 20:20:07.136568 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:07.136542    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-407eaab7-4tfp2"]
Jul 14 20:20:07.218238 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:07.218204    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e55e36-5c7d-4b54-934d-a04fd241df72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-407eaab7-4tfp2\" (UID: \"f8e55e36-5c7d-4b54-934d-a04fd241df72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-407eaab7-4tfp2"
Jul 14 20:20:07.319776 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:07.319735    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e55e36-5c7d-4b54-934d-a04fd241df72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-407eaab7-4tfp2\" (UID: \"f8e55e36-5c7d-4b54-934d-a04fd241df72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-407eaab7-4tfp2"
Jul 14 20:20:07.320531 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:07.320506    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e55e36-5c7d-4b54-934d-a04fd241df72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-407eaab7-4tfp2\" (UID: \"f8e55e36-5c7d-4b54-934d-a04fd241df72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-407eaab7-4tfp2"
Jul 14 20:20:07.440305 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:07.440213    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-407eaab7-4tfp2"
Jul 14 20:20:07.577103 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:07.577076    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-407eaab7-4tfp2"]
Jul 14 20:20:07.579569 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:20:07.579542    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e55e36_5c7d_4b54_934d_a04fd241df72.slice/crio-e67db3240bc75db1fd83eff7727200457cd0c82d5267f90b7a413502f36fd02e WatchSource:0}: Error finding container e67db3240bc75db1fd83eff7727200457cd0c82d5267f90b7a413502f36fd02e: Status 404 returned error can't find the container with id e67db3240bc75db1fd83eff7727200457cd0c82d5267f90b7a413502f36fd02e
Jul 14 20:20:08.080110 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:08.080018    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-407eaab7-4tfp2" event={"ID":"f8e55e36-5c7d-4b54-934d-a04fd241df72","Type":"ContainerStarted","Data":"3208d4e0c70253e5f39cc31c5a20d0d50c88e0e5c369cd995b3a6faf455a3c25"}
Jul 14 20:20:08.080110 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:08.080072    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-407eaab7-4tfp2" event={"ID":"f8e55e36-5c7d-4b54-934d-a04fd241df72","Type":"ContainerStarted","Data":"e67db3240bc75db1fd83eff7727200457cd0c82d5267f90b7a413502f36fd02e"}
Jul 14 20:20:08.095523 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:08.095479    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-407eaab7-4tfp2" podStartSLOduration=1.095463365 podStartE2EDuration="1.095463365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:20:08.094824138 +0000 UTC m=+1960.695127841" watchObservedRunningTime="2026-07-14 20:20:08.095463365 +0000 UTC m=+1960.695767046"
Jul 14 20:20:09.086720 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:09.086688    2560 generic.go:356] "Generic (PLEG): container finished" podID="f8e55e36-5c7d-4b54-934d-a04fd241df72" containerID="3208d4e0c70253e5f39cc31c5a20d0d50c88e0e5c369cd995b3a6faf455a3c25" exitCode=0
Jul 14 20:20:09.087088 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:09.086774    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-407eaab7-4tfp2" event={"ID":"f8e55e36-5c7d-4b54-934d-a04fd241df72","Type":"ContainerDied","Data":"3208d4e0c70253e5f39cc31c5a20d0d50c88e0e5c369cd995b3a6faf455a3c25"}
Jul 14 20:20:10.222967 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:10.222939    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-407eaab7-4tfp2"
Jul 14 20:20:10.348133 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:10.348045    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e55e36-5c7d-4b54-934d-a04fd241df72-maas-api-service-ca\") pod \"f8e55e36-5c7d-4b54-934d-a04fd241df72\" (UID: \"f8e55e36-5c7d-4b54-934d-a04fd241df72\") "
Jul 14 20:20:10.348457 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:10.348430    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e55e36-5c7d-4b54-934d-a04fd241df72-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8e55e36-5c7d-4b54-934d-a04fd241df72" (UID: "f8e55e36-5c7d-4b54-934d-a04fd241df72"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 20:20:10.448802 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:10.448768    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e55e36-5c7d-4b54-934d-a04fd241df72-maas-api-service-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:20:11.098066 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:11.098037    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-407eaab7-4tfp2"
Jul 14 20:20:11.098066 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:11.098064    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-407eaab7-4tfp2" event={"ID":"f8e55e36-5c7d-4b54-934d-a04fd241df72","Type":"ContainerDied","Data":"e67db3240bc75db1fd83eff7727200457cd0c82d5267f90b7a413502f36fd02e"}
Jul 14 20:20:11.098352 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:11.098110    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e67db3240bc75db1fd83eff7727200457cd0c82d5267f90b7a413502f36fd02e"
Jul 14 20:20:17.653682 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:17.653646    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr"]
Jul 14 20:20:17.654232 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:17.653936    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr" podUID="a8f4dd92-8fce-48b7-a88d-17cb03152f1e" containerName="maas-api" containerID="cri-o://9474c4316288071dbec51a0e07c4749efb88ee5f7bce78e08c47e2aa848b9d25" gracePeriod=30
Jul 14 20:20:17.907187 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:17.907097    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr"
Jul 14 20:20:18.022148 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:18.022109    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8f4dd92-8fce-48b7-a88d-17cb03152f1e-maas-api-tls\") pod \"a8f4dd92-8fce-48b7-a88d-17cb03152f1e\" (UID: \"a8f4dd92-8fce-48b7-a88d-17cb03152f1e\") "
Jul 14 20:20:18.022346 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:18.022194    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f4dd92-8fce-48b7-a88d-17cb03152f1e-openshift-service-ca\") pod \"a8f4dd92-8fce-48b7-a88d-17cb03152f1e\" (UID: \"a8f4dd92-8fce-48b7-a88d-17cb03152f1e\") "
Jul 14 20:20:18.022346 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:18.022334    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqs4n\" (UniqueName: \"kubernetes.io/projected/a8f4dd92-8fce-48b7-a88d-17cb03152f1e-kube-api-access-rqs4n\") pod \"a8f4dd92-8fce-48b7-a88d-17cb03152f1e\" (UID: \"a8f4dd92-8fce-48b7-a88d-17cb03152f1e\") "
Jul 14 20:20:18.022630 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:18.022589    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f4dd92-8fce-48b7-a88d-17cb03152f1e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a8f4dd92-8fce-48b7-a88d-17cb03152f1e" (UID: "a8f4dd92-8fce-48b7-a88d-17cb03152f1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 20:20:18.024461 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:18.024429    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8f4dd92-8fce-48b7-a88d-17cb03152f1e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a8f4dd92-8fce-48b7-a88d-17cb03152f1e" (UID: "a8f4dd92-8fce-48b7-a88d-17cb03152f1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 20:20:18.024617 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:18.024504    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f4dd92-8fce-48b7-a88d-17cb03152f1e-kube-api-access-rqs4n" (OuterVolumeSpecName: "kube-api-access-rqs4n") pod "a8f4dd92-8fce-48b7-a88d-17cb03152f1e" (UID: "a8f4dd92-8fce-48b7-a88d-17cb03152f1e"). InnerVolumeSpecName "kube-api-access-rqs4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 20:20:18.123690 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:18.123658    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqs4n\" (UniqueName: \"kubernetes.io/projected/a8f4dd92-8fce-48b7-a88d-17cb03152f1e-kube-api-access-rqs4n\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:20:18.123690 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:18.123689    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8f4dd92-8fce-48b7-a88d-17cb03152f1e-maas-api-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:20:18.123690 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:18.123699    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f4dd92-8fce-48b7-a88d-17cb03152f1e-openshift-service-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:20:18.143895 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:18.143824    2560 generic.go:356] "Generic (PLEG): container finished" podID="a8f4dd92-8fce-48b7-a88d-17cb03152f1e" containerID="9474c4316288071dbec51a0e07c4749efb88ee5f7bce78e08c47e2aa848b9d25" exitCode=0
Jul 14 20:20:18.143895 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:18.143862    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr" event={"ID":"a8f4dd92-8fce-48b7-a88d-17cb03152f1e","Type":"ContainerDied","Data":"9474c4316288071dbec51a0e07c4749efb88ee5f7bce78e08c47e2aa848b9d25"}
Jul 14 20:20:18.143895 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:18.143873    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr"
Jul 14 20:20:18.144139 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:18.143933    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr" event={"ID":"a8f4dd92-8fce-48b7-a88d-17cb03152f1e","Type":"ContainerDied","Data":"a79773fe64b727dafbb382aaaf2fb31074db86dbd3f273f26bd6d01fad503f3c"}
Jul 14 20:20:18.144139 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:18.143954    2560 scope.go:117] "RemoveContainer" containerID="9474c4316288071dbec51a0e07c4749efb88ee5f7bce78e08c47e2aa848b9d25"
Jul 14 20:20:18.153085 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:18.153066    2560 scope.go:117] "RemoveContainer" containerID="9474c4316288071dbec51a0e07c4749efb88ee5f7bce78e08c47e2aa848b9d25"
Jul 14 20:20:18.153363 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:20:18.153345    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9474c4316288071dbec51a0e07c4749efb88ee5f7bce78e08c47e2aa848b9d25\": container with ID starting with 9474c4316288071dbec51a0e07c4749efb88ee5f7bce78e08c47e2aa848b9d25 not found: ID does not exist" containerID="9474c4316288071dbec51a0e07c4749efb88ee5f7bce78e08c47e2aa848b9d25"
Jul 14 20:20:18.153412 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:18.153373    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9474c4316288071dbec51a0e07c4749efb88ee5f7bce78e08c47e2aa848b9d25"} err="failed to get container status \"9474c4316288071dbec51a0e07c4749efb88ee5f7bce78e08c47e2aa848b9d25\": rpc error: code = NotFound desc = could not find container \"9474c4316288071dbec51a0e07c4749efb88ee5f7bce78e08c47e2aa848b9d25\": container with ID starting with 9474c4316288071dbec51a0e07c4749efb88ee5f7bce78e08c47e2aa848b9d25 not found: ID does not exist"
Jul 14 20:20:18.165629 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:18.165570    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr"]
Jul 14 20:20:18.169850 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:18.169830    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-407eaab7-54cf4f78dc-9rrvr"]
Jul 14 20:20:19.889304 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:19.889269    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8f4dd92-8fce-48b7-a88d-17cb03152f1e" path="/var/lib/kubelet/pods/a8f4dd92-8fce-48b7-a88d-17cb03152f1e/volumes"
Jul 14 20:20:55.710362 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:55.710324    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm"]
Jul 14 20:20:55.710902 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:55.710885    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e55e36-5c7d-4b54-934d-a04fd241df72" containerName="revoke-keys"
Jul 14 20:20:55.710951 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:55.710908    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e55e36-5c7d-4b54-934d-a04fd241df72" containerName="revoke-keys"
Jul 14 20:20:55.710951 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:55.710940    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8f4dd92-8fce-48b7-a88d-17cb03152f1e" containerName="maas-api"
Jul 14 20:20:55.711015 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:55.710949    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8f4dd92-8fce-48b7-a88d-17cb03152f1e" containerName="maas-api"
Jul 14 20:20:55.711068 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:55.711056    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8e55e36-5c7d-4b54-934d-a04fd241df72" containerName="revoke-keys"
Jul 14 20:20:55.711110 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:55.711072    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8f4dd92-8fce-48b7-a88d-17cb03152f1e" containerName="maas-api"
Jul 14 20:20:55.714219 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:55.714201    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm"
Jul 14 20:20:55.716859 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:55.716836    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-cbbc800e-serving-cert\""
Jul 14 20:20:55.716987 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:55.716864    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gfkms\""
Jul 14 20:20:55.725153 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:55.725131    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm"]
Jul 14 20:20:55.854390 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:55.854351    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6b8768-fceb-4018-a5cf-0d8594e75817-openshift-service-ca\") pod \"maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm\" (UID: \"aa6b8768-fceb-4018-a5cf-0d8594e75817\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm"
Jul 14 20:20:55.854607 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:55.854397    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cq4n\" (UniqueName: \"kubernetes.io/projected/aa6b8768-fceb-4018-a5cf-0d8594e75817-kube-api-access-9cq4n\") pod \"maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm\" (UID: \"aa6b8768-fceb-4018-a5cf-0d8594e75817\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm"
Jul 14 20:20:55.854607 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:55.854516    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa6b8768-fceb-4018-a5cf-0d8594e75817-maas-api-tls\") pod \"maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm\" (UID: \"aa6b8768-fceb-4018-a5cf-0d8594e75817\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm"
Jul 14 20:20:55.955014 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:55.954976    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa6b8768-fceb-4018-a5cf-0d8594e75817-maas-api-tls\") pod \"maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm\" (UID: \"aa6b8768-fceb-4018-a5cf-0d8594e75817\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm"
Jul 14 20:20:55.955225 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:55.955071    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6b8768-fceb-4018-a5cf-0d8594e75817-openshift-service-ca\") pod \"maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm\" (UID: \"aa6b8768-fceb-4018-a5cf-0d8594e75817\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm"
Jul 14 20:20:55.955225 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:55.955108    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cq4n\" (UniqueName: \"kubernetes.io/projected/aa6b8768-fceb-4018-a5cf-0d8594e75817-kube-api-access-9cq4n\") pod \"maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm\" (UID: \"aa6b8768-fceb-4018-a5cf-0d8594e75817\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm"
Jul 14 20:20:55.955225 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:20:55.955125    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-serving-cert: secret "maas-api-e2e-mt-cbbc800e-serving-cert" not found
Jul 14 20:20:55.955225 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:20:55.955213    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa6b8768-fceb-4018-a5cf-0d8594e75817-maas-api-tls podName:aa6b8768-fceb-4018-a5cf-0d8594e75817 nodeName:}" failed. No retries permitted until 2026-07-14 20:20:56.455195677 +0000 UTC m=+2009.055499357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aa6b8768-fceb-4018-a5cf-0d8594e75817-maas-api-tls") pod "maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm" (UID: "aa6b8768-fceb-4018-a5cf-0d8594e75817") : secret "maas-api-e2e-mt-cbbc800e-serving-cert" not found
Jul 14 20:20:55.955690 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:55.955671    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6b8768-fceb-4018-a5cf-0d8594e75817-openshift-service-ca\") pod \"maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm\" (UID: \"aa6b8768-fceb-4018-a5cf-0d8594e75817\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm"
Jul 14 20:20:55.964856 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:55.964797    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cq4n\" (UniqueName: \"kubernetes.io/projected/aa6b8768-fceb-4018-a5cf-0d8594e75817-kube-api-access-9cq4n\") pod \"maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm\" (UID: \"aa6b8768-fceb-4018-a5cf-0d8594e75817\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm"
Jul 14 20:20:56.466193 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:56.466141    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa6b8768-fceb-4018-a5cf-0d8594e75817-maas-api-tls\") pod \"maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm\" (UID: \"aa6b8768-fceb-4018-a5cf-0d8594e75817\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm"
Jul 14 20:20:56.469265 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:56.469241    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa6b8768-fceb-4018-a5cf-0d8594e75817-maas-api-tls\") pod \"maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm\" (UID: \"aa6b8768-fceb-4018-a5cf-0d8594e75817\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm"
Jul 14 20:20:56.627636 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:56.627598    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm"
Jul 14 20:20:56.770865 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:56.770750    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm"]
Jul 14 20:20:56.773101 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:20:56.773077    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa6b8768_fceb_4018_a5cf_0d8594e75817.slice/crio-a3ebac51efd57abc45cae979f0bf39f55317cd6d398ca5b9439f4af106d54b67 WatchSource:0}: Error finding container a3ebac51efd57abc45cae979f0bf39f55317cd6d398ca5b9439f4af106d54b67: Status 404 returned error can't find the container with id a3ebac51efd57abc45cae979f0bf39f55317cd6d398ca5b9439f4af106d54b67
Jul 14 20:20:57.341273 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:57.341241    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm" event={"ID":"aa6b8768-fceb-4018-a5cf-0d8594e75817","Type":"ContainerStarted","Data":"6c6bbae4c479a7048fe529ab570ad7490cbb39641690e34364601a5de53c43cf"}
Jul 14 20:20:57.341447 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:57.341297    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm" event={"ID":"aa6b8768-fceb-4018-a5cf-0d8594e75817","Type":"ContainerStarted","Data":"a3ebac51efd57abc45cae979f0bf39f55317cd6d398ca5b9439f4af106d54b67"}
Jul 14 20:20:57.341447 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:57.341379    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm"
Jul 14 20:20:57.362088 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:20:57.362029    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm" podStartSLOduration=2.001140688 podStartE2EDuration="2.362013512s" totalImagesPullingTime="360.872824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:20:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:20:57.360233076 +0000 UTC m=+2009.960536780" watchObservedRunningTime="2026-07-14 20:20:57.362013512 +0000 UTC m=+2009.962317214"
Jul 14 20:21:03.351859 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:03.351826    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm"
Jul 14 20:21:06.549751 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:06.549718    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"]
Jul 14 20:21:06.553377 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:06.553360    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"
Jul 14 20:21:06.555778 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:06.555755    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cbbc800e\"/\"openshift-service-ca.crt\""
Jul 14 20:21:06.557074 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:06.557049    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cbbc800e\"/\"default-dockercfg-vfzg2\""
Jul 14 20:21:06.557203 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:06.557057    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cbbc800e\"/\"e2e-lifecycle-model-cbbc800e-kserve-self-signed-certs\""
Jul 14 20:21:06.557203 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:06.557117    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cbbc800e\"/\"kube-root-ca.crt\""
Jul 14 20:21:06.561044 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:06.561022    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"]
Jul 14 20:21:06.660988 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:06.660948    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e1aa32-0743-4150-b029-8799184c1a22-tls-certs\") pod \"e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7\" (UID: \"a0e1aa32-0743-4150-b029-8799184c1a22\") " pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"
Jul 14 20:21:06.660988 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:06.660988    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrsm9\" (UniqueName: \"kubernetes.io/projected/a0e1aa32-0743-4150-b029-8799184c1a22-kube-api-access-rrsm9\") pod \"e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7\" (UID: \"a0e1aa32-0743-4150-b029-8799184c1a22\") " pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"
Jul 14 20:21:06.661235 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:06.661012    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e1aa32-0743-4150-b029-8799184c1a22-home\") pod \"e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7\" (UID: \"a0e1aa32-0743-4150-b029-8799184c1a22\") " pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"
Jul 14 20:21:06.661235 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:06.661118    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e1aa32-0743-4150-b029-8799184c1a22-model-cache\") pod \"e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7\" (UID: \"a0e1aa32-0743-4150-b029-8799184c1a22\") " pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"
Jul 14 20:21:06.661235 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:06.661150    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e1aa32-0743-4150-b029-8799184c1a22-tmp-dir\") pod \"e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7\" (UID: \"a0e1aa32-0743-4150-b029-8799184c1a22\") " pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"
Jul 14 20:21:06.661235 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:06.661191    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e1aa32-0743-4150-b029-8799184c1a22-dshm\") pod \"e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7\" (UID: \"a0e1aa32-0743-4150-b029-8799184c1a22\") " pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"
Jul 14 20:21:06.762588 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:06.762547    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e1aa32-0743-4150-b029-8799184c1a22-tls-certs\") pod \"e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7\" (UID: \"a0e1aa32-0743-4150-b029-8799184c1a22\") " pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"
Jul 14 20:21:06.762588 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:06.762591    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrsm9\" (UniqueName: \"kubernetes.io/projected/a0e1aa32-0743-4150-b029-8799184c1a22-kube-api-access-rrsm9\") pod \"e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7\" (UID: \"a0e1aa32-0743-4150-b029-8799184c1a22\") " pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"
Jul 14 20:21:06.762813 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:06.762614    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e1aa32-0743-4150-b029-8799184c1a22-home\") pod \"e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7\" (UID: \"a0e1aa32-0743-4150-b029-8799184c1a22\") " pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"
Jul 14 20:21:06.762813 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:06.762662    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e1aa32-0743-4150-b029-8799184c1a22-model-cache\") pod \"e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7\" (UID: \"a0e1aa32-0743-4150-b029-8799184c1a22\") " pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"
Jul 14 20:21:06.762813 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:06.762682    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e1aa32-0743-4150-b029-8799184c1a22-tmp-dir\") pod \"e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7\" (UID: \"a0e1aa32-0743-4150-b029-8799184c1a22\") " pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"
Jul 14 20:21:06.762813 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:06.762698    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e1aa32-0743-4150-b029-8799184c1a22-dshm\") pod \"e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7\" (UID: \"a0e1aa32-0743-4150-b029-8799184c1a22\") " pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"
Jul 14 20:21:06.763136 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:06.763112    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e1aa32-0743-4150-b029-8799184c1a22-model-cache\") pod \"e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7\" (UID: \"a0e1aa32-0743-4150-b029-8799184c1a22\") " pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"
Jul 14 20:21:06.763256 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:06.763155    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e1aa32-0743-4150-b029-8799184c1a22-tmp-dir\") pod \"e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7\" (UID: \"a0e1aa32-0743-4150-b029-8799184c1a22\") " pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"
Jul 14 20:21:06.763256 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:06.763165    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e1aa32-0743-4150-b029-8799184c1a22-home\") pod \"e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7\" (UID: \"a0e1aa32-0743-4150-b029-8799184c1a22\") " pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"
Jul 14 20:21:06.765097 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:06.765073    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e1aa32-0743-4150-b029-8799184c1a22-dshm\") pod \"e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7\" (UID: \"a0e1aa32-0743-4150-b029-8799184c1a22\") " pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"
Jul 14 20:21:06.765271 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:06.765254    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e1aa32-0743-4150-b029-8799184c1a22-tls-certs\") pod \"e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7\" (UID: \"a0e1aa32-0743-4150-b029-8799184c1a22\") " pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"
Jul 14 20:21:06.772024 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:06.771999    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrsm9\" (UniqueName: \"kubernetes.io/projected/a0e1aa32-0743-4150-b029-8799184c1a22-kube-api-access-rrsm9\") pod \"e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7\" (UID: \"a0e1aa32-0743-4150-b029-8799184c1a22\") " pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"
Jul 14 20:21:06.864110 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:06.864015    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"
Jul 14 20:21:07.006391 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:07.006364    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"]
Jul 14 20:21:07.008827 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:21:07.008798    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0e1aa32_0743_4150_b029_8799184c1a22.slice/crio-e2d42cd7aee192a98e6147afeb4c5ce9a8129271b9cb43a739a95389cce7aec9 WatchSource:0}: Error finding container e2d42cd7aee192a98e6147afeb4c5ce9a8129271b9cb43a739a95389cce7aec9: Status 404 returned error can't find the container with id e2d42cd7aee192a98e6147afeb4c5ce9a8129271b9cb43a739a95389cce7aec9
Jul 14 20:21:07.394877 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:07.394846    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7" event={"ID":"a0e1aa32-0743-4150-b029-8799184c1a22","Type":"ContainerStarted","Data":"4b6cb530a0476699f4e6cd31eef2a5ce10edef06de9c3ebab5c46c823158a23f"}
Jul 14 20:21:07.395050 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:07.394889    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7" event={"ID":"a0e1aa32-0743-4150-b029-8799184c1a22","Type":"ContainerStarted","Data":"e2d42cd7aee192a98e6147afeb4c5ce9a8129271b9cb43a739a95389cce7aec9"}
Jul 14 20:21:07.425736 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:07.425679    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7" podStartSLOduration=1.425660551 podStartE2EDuration="1.425660551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:21:07.422673829 +0000 UTC m=+2020.022977543" watchObservedRunningTime="2026-07-14 20:21:07.425660551 +0000 UTC m=+2020.025964254"
Jul 14 20:21:16.864655 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:16.864617    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"
Jul 14 20:21:16.865142 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:16.864674    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"
Jul 14 20:21:16.877487 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:16.877461    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"
Jul 14 20:21:17.467901 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:17.467868    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"
Jul 14 20:21:28.074366 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:28.074338    2560 scope.go:117] "RemoveContainer" containerID="6ef449b70a3e853ec23781a54f3726082e85f96d21c6c95c9247b9ac23591a4f"
Jul 14 20:21:28.082712 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:28.082692    2560 scope.go:117] "RemoveContainer" containerID="78e129ebd27a543c15c2254f920af8a172852ccd24632195f40762709e84eb3a"
Jul 14 20:21:38.717935 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:38.717880    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm"]
Jul 14 20:21:38.720019 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:38.719950    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm" podUID="aa6b8768-fceb-4018-a5cf-0d8594e75817" containerName="maas-api" containerID="cri-o://6c6bbae4c479a7048fe529ab570ad7490cbb39641690e34364601a5de53c43cf" gracePeriod=30
Jul 14 20:21:38.971877 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:38.971814    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm"
Jul 14 20:21:39.051248 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:39.051208    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cq4n\" (UniqueName: \"kubernetes.io/projected/aa6b8768-fceb-4018-a5cf-0d8594e75817-kube-api-access-9cq4n\") pod \"aa6b8768-fceb-4018-a5cf-0d8594e75817\" (UID: \"aa6b8768-fceb-4018-a5cf-0d8594e75817\") "
Jul 14 20:21:39.051425 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:39.051297    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa6b8768-fceb-4018-a5cf-0d8594e75817-maas-api-tls\") pod \"aa6b8768-fceb-4018-a5cf-0d8594e75817\" (UID: \"aa6b8768-fceb-4018-a5cf-0d8594e75817\") "
Jul 14 20:21:39.051425 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:39.051367    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6b8768-fceb-4018-a5cf-0d8594e75817-openshift-service-ca\") pod \"aa6b8768-fceb-4018-a5cf-0d8594e75817\" (UID: \"aa6b8768-fceb-4018-a5cf-0d8594e75817\") "
Jul 14 20:21:39.051734 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:39.051705    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa6b8768-fceb-4018-a5cf-0d8594e75817-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aa6b8768-fceb-4018-a5cf-0d8594e75817" (UID: "aa6b8768-fceb-4018-a5cf-0d8594e75817"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 20:21:39.053509 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:39.053481    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa6b8768-fceb-4018-a5cf-0d8594e75817-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aa6b8768-fceb-4018-a5cf-0d8594e75817" (UID: "aa6b8768-fceb-4018-a5cf-0d8594e75817"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 20:21:39.053611 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:39.053529    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa6b8768-fceb-4018-a5cf-0d8594e75817-kube-api-access-9cq4n" (OuterVolumeSpecName: "kube-api-access-9cq4n") pod "aa6b8768-fceb-4018-a5cf-0d8594e75817" (UID: "aa6b8768-fceb-4018-a5cf-0d8594e75817"). InnerVolumeSpecName "kube-api-access-9cq4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 20:21:39.152998 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:39.152960    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6b8768-fceb-4018-a5cf-0d8594e75817-openshift-service-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:21:39.152998 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:39.152992    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cq4n\" (UniqueName: \"kubernetes.io/projected/aa6b8768-fceb-4018-a5cf-0d8594e75817-kube-api-access-9cq4n\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:21:39.152998 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:39.153002    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa6b8768-fceb-4018-a5cf-0d8594e75817-maas-api-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:21:39.568098 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:39.568059    2560 generic.go:356] "Generic (PLEG): container finished" podID="aa6b8768-fceb-4018-a5cf-0d8594e75817" containerID="6c6bbae4c479a7048fe529ab570ad7490cbb39641690e34364601a5de53c43cf" exitCode=0
Jul 14 20:21:39.568098 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:39.568100    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm"
Jul 14 20:21:39.568329 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:39.568119    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm" event={"ID":"aa6b8768-fceb-4018-a5cf-0d8594e75817","Type":"ContainerDied","Data":"6c6bbae4c479a7048fe529ab570ad7490cbb39641690e34364601a5de53c43cf"}
Jul 14 20:21:39.568329 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:39.568191    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm" event={"ID":"aa6b8768-fceb-4018-a5cf-0d8594e75817","Type":"ContainerDied","Data":"a3ebac51efd57abc45cae979f0bf39f55317cd6d398ca5b9439f4af106d54b67"}
Jul 14 20:21:39.568329 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:39.568218    2560 scope.go:117] "RemoveContainer" containerID="6c6bbae4c479a7048fe529ab570ad7490cbb39641690e34364601a5de53c43cf"
Jul 14 20:21:39.577908 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:39.577891    2560 scope.go:117] "RemoveContainer" containerID="6c6bbae4c479a7048fe529ab570ad7490cbb39641690e34364601a5de53c43cf"
Jul 14 20:21:39.578218 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:21:39.578195    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c6bbae4c479a7048fe529ab570ad7490cbb39641690e34364601a5de53c43cf\": container with ID starting with 6c6bbae4c479a7048fe529ab570ad7490cbb39641690e34364601a5de53c43cf not found: ID does not exist" containerID="6c6bbae4c479a7048fe529ab570ad7490cbb39641690e34364601a5de53c43cf"
Jul 14 20:21:39.578280 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:39.578231    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c6bbae4c479a7048fe529ab570ad7490cbb39641690e34364601a5de53c43cf"} err="failed to get container status \"6c6bbae4c479a7048fe529ab570ad7490cbb39641690e34364601a5de53c43cf\": rpc error: code = NotFound desc = could not find container \"6c6bbae4c479a7048fe529ab570ad7490cbb39641690e34364601a5de53c43cf\": container with ID starting with 6c6bbae4c479a7048fe529ab570ad7490cbb39641690e34364601a5de53c43cf not found: ID does not exist"
Jul 14 20:21:39.590068 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:39.590043    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm"]
Jul 14 20:21:39.594573 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:39.594549    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cbbc800e-5988cf64f5-rd6gm"]
Jul 14 20:21:39.890094 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:39.890018    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa6b8768-fceb-4018-a5cf-0d8594e75817" path="/var/lib/kubelet/pods/aa6b8768-fceb-4018-a5cf-0d8594e75817/volumes"
Jul 14 20:21:48.357210 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:21:48.357085    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"]
Jul 14 20:22:03.378263 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.378201    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7" podUID="a0e1aa32-0743-4150-b029-8799184c1a22" containerName="main" containerID="cri-o://4b6cb530a0476699f4e6cd31eef2a5ce10edef06de9c3ebab5c46c823158a23f" gracePeriod=45
Jul 14 20:22:03.641358 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.641294    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"
Jul 14 20:22:03.682306 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.682272    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrsm9\" (UniqueName: \"kubernetes.io/projected/a0e1aa32-0743-4150-b029-8799184c1a22-kube-api-access-rrsm9\") pod \"a0e1aa32-0743-4150-b029-8799184c1a22\" (UID: \"a0e1aa32-0743-4150-b029-8799184c1a22\") "
Jul 14 20:22:03.682468 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.682360    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e1aa32-0743-4150-b029-8799184c1a22-model-cache\") pod \"a0e1aa32-0743-4150-b029-8799184c1a22\" (UID: \"a0e1aa32-0743-4150-b029-8799184c1a22\") "
Jul 14 20:22:03.682468 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.682402    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e1aa32-0743-4150-b029-8799184c1a22-tmp-dir\") pod \"a0e1aa32-0743-4150-b029-8799184c1a22\" (UID: \"a0e1aa32-0743-4150-b029-8799184c1a22\") "
Jul 14 20:22:03.682468 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.682460    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e1aa32-0743-4150-b029-8799184c1a22-home\") pod \"a0e1aa32-0743-4150-b029-8799184c1a22\" (UID: \"a0e1aa32-0743-4150-b029-8799184c1a22\") "
Jul 14 20:22:03.682651 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.682539    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e1aa32-0743-4150-b029-8799184c1a22-tls-certs\") pod \"a0e1aa32-0743-4150-b029-8799184c1a22\" (UID: \"a0e1aa32-0743-4150-b029-8799184c1a22\") "
Jul 14 20:22:03.682651 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.682583    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e1aa32-0743-4150-b029-8799184c1a22-dshm\") pod \"a0e1aa32-0743-4150-b029-8799184c1a22\" (UID: \"a0e1aa32-0743-4150-b029-8799184c1a22\") "
Jul 14 20:22:03.682765 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.682655    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e1aa32-0743-4150-b029-8799184c1a22-model-cache" (OuterVolumeSpecName: "model-cache") pod "a0e1aa32-0743-4150-b029-8799184c1a22" (UID: "a0e1aa32-0743-4150-b029-8799184c1a22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 20:22:03.682765 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.682693    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e1aa32-0743-4150-b029-8799184c1a22-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a0e1aa32-0743-4150-b029-8799184c1a22" (UID: "a0e1aa32-0743-4150-b029-8799184c1a22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 20:22:03.682875 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.682776    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e1aa32-0743-4150-b029-8799184c1a22-home" (OuterVolumeSpecName: "home") pod "a0e1aa32-0743-4150-b029-8799184c1a22" (UID: "a0e1aa32-0743-4150-b029-8799184c1a22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 20:22:03.682932 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.682910    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e1aa32-0743-4150-b029-8799184c1a22-home\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:22:03.682992 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.682931    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e1aa32-0743-4150-b029-8799184c1a22-model-cache\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:22:03.682992 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.682945    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e1aa32-0743-4150-b029-8799184c1a22-tmp-dir\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:22:03.684988 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.684956    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0e1aa32-0743-4150-b029-8799184c1a22-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a0e1aa32-0743-4150-b029-8799184c1a22" (UID: "a0e1aa32-0743-4150-b029-8799184c1a22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 20:22:03.685105 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.684993    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e1aa32-0743-4150-b029-8799184c1a22-dshm" (OuterVolumeSpecName: "dshm") pod "a0e1aa32-0743-4150-b029-8799184c1a22" (UID: "a0e1aa32-0743-4150-b029-8799184c1a22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 20:22:03.685147 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.685114    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e1aa32-0743-4150-b029-8799184c1a22-kube-api-access-rrsm9" (OuterVolumeSpecName: "kube-api-access-rrsm9") pod "a0e1aa32-0743-4150-b029-8799184c1a22" (UID: "a0e1aa32-0743-4150-b029-8799184c1a22"). InnerVolumeSpecName "kube-api-access-rrsm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 20:22:03.698662 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.698635    2560 generic.go:356] "Generic (PLEG): container finished" podID="a0e1aa32-0743-4150-b029-8799184c1a22" containerID="4b6cb530a0476699f4e6cd31eef2a5ce10edef06de9c3ebab5c46c823158a23f" exitCode=0
Jul 14 20:22:03.698798 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.698683    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"
Jul 14 20:22:03.698798 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.698716    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7" event={"ID":"a0e1aa32-0743-4150-b029-8799184c1a22","Type":"ContainerDied","Data":"4b6cb530a0476699f4e6cd31eef2a5ce10edef06de9c3ebab5c46c823158a23f"}
Jul 14 20:22:03.698798 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.698766    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7" event={"ID":"a0e1aa32-0743-4150-b029-8799184c1a22","Type":"ContainerDied","Data":"e2d42cd7aee192a98e6147afeb4c5ce9a8129271b9cb43a739a95389cce7aec9"}
Jul 14 20:22:03.698798 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.698789    2560 scope.go:117] "RemoveContainer" containerID="4b6cb530a0476699f4e6cd31eef2a5ce10edef06de9c3ebab5c46c823158a23f"
Jul 14 20:22:03.708441 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.708426    2560 scope.go:117] "RemoveContainer" containerID="4b6cb530a0476699f4e6cd31eef2a5ce10edef06de9c3ebab5c46c823158a23f"
Jul 14 20:22:03.708743 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:22:03.708718    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b6cb530a0476699f4e6cd31eef2a5ce10edef06de9c3ebab5c46c823158a23f\": container with ID starting with 4b6cb530a0476699f4e6cd31eef2a5ce10edef06de9c3ebab5c46c823158a23f not found: ID does not exist" containerID="4b6cb530a0476699f4e6cd31eef2a5ce10edef06de9c3ebab5c46c823158a23f"
Jul 14 20:22:03.708851 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.708746    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b6cb530a0476699f4e6cd31eef2a5ce10edef06de9c3ebab5c46c823158a23f"} err="failed to get container status \"4b6cb530a0476699f4e6cd31eef2a5ce10edef06de9c3ebab5c46c823158a23f\": rpc error: code = NotFound desc = could not find container \"4b6cb530a0476699f4e6cd31eef2a5ce10edef06de9c3ebab5c46c823158a23f\": container with ID starting with 4b6cb530a0476699f4e6cd31eef2a5ce10edef06de9c3ebab5c46c823158a23f not found: ID does not exist"
Jul 14 20:22:03.720068 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.720041    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"]
Jul 14 20:22:03.726226 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.726201    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-cbbc800e/e2e-lifecycle-model-cbbc800e-kserve-678b7f6bff-hnjn7"]
Jul 14 20:22:03.783937 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.783899    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e1aa32-0743-4150-b029-8799184c1a22-tls-certs\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:22:03.783937 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.783927    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e1aa32-0743-4150-b029-8799184c1a22-dshm\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:22:03.783937 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.783937    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrsm9\" (UniqueName: \"kubernetes.io/projected/a0e1aa32-0743-4150-b029-8799184c1a22-kube-api-access-rrsm9\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:22:03.890070 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:03.890035    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0e1aa32-0743-4150-b029-8799184c1a22" path="/var/lib/kubelet/pods/a0e1aa32-0743-4150-b029-8799184c1a22/volumes"
Jul 14 20:22:27.979677 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:27.979646    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tzr6_4a36af4c-9eac-465f-9a0e-d65a382530f7/ovn-acl-logging/0.log"
Jul 14 20:22:27.989723 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:27.989701    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tzr6_4a36af4c-9eac-465f-9a0e-d65a382530f7/ovn-acl-logging/0.log"
Jul 14 20:22:28.115026 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:22:28.114997    2560 scope.go:117] "RemoveContainer" containerID="f8c6fc32a7b99e0bad4f4c90433b1fa61776d4706a563988e039d1702b232b04"
Jul 14 20:23:28.142075 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:23:28.141999    2560 scope.go:117] "RemoveContainer" containerID="687896a4315cba84db072b41f903a7c937ba55ca1296cf3a5d4bfd8c81de59a7"
Jul 14 20:23:53.674219 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:23:53.674184    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49"]
Jul 14 20:23:53.674832 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:23:53.674728    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa6b8768-fceb-4018-a5cf-0d8594e75817" containerName="maas-api"
Jul 14 20:23:53.674832 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:23:53.674748    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa6b8768-fceb-4018-a5cf-0d8594e75817" containerName="maas-api"
Jul 14 20:23:53.674832 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:23:53.674790    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0e1aa32-0743-4150-b029-8799184c1a22" containerName="main"
Jul 14 20:23:53.674832 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:23:53.674798    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0e1aa32-0743-4150-b029-8799184c1a22" containerName="main"
Jul 14 20:23:53.675051 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:23:53.674886    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0e1aa32-0743-4150-b029-8799184c1a22" containerName="main"
Jul 14 20:23:53.675051 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:23:53.674904    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa6b8768-fceb-4018-a5cf-0d8594e75817" containerName="maas-api"
Jul 14 20:23:53.678058 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:23:53.678039    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49"
Jul 14 20:23:53.680535 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:23:53.680518    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-8d5be0-serving-cert\""
Jul 14 20:23:53.680635 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:23:53.680559    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gfkms\""
Jul 14 20:23:53.691479 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:23:53.691456    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49"]
Jul 14 20:23:53.777924 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:23:53.777891    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e097f0-7d33-4930-a83a-8077a0465fda-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49\" (UID: \"b7e097f0-7d33-4930-a83a-8077a0465fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49"
Jul 14 20:23:53.777924 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:23:53.777933    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxrkl\" (UniqueName: \"kubernetes.io/projected/b7e097f0-7d33-4930-a83a-8077a0465fda-kube-api-access-zxrkl\") pod \"maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49\" (UID: \"b7e097f0-7d33-4930-a83a-8077a0465fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49"
Jul 14 20:23:53.778140 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:23:53.778041    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7e097f0-7d33-4930-a83a-8077a0465fda-maas-api-tls\") pod \"maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49\" (UID: \"b7e097f0-7d33-4930-a83a-8077a0465fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49"
Jul 14 20:23:53.878933 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:23:53.878897    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e097f0-7d33-4930-a83a-8077a0465fda-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49\" (UID: \"b7e097f0-7d33-4930-a83a-8077a0465fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49"
Jul 14 20:23:53.878933 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:23:53.878933    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxrkl\" (UniqueName: \"kubernetes.io/projected/b7e097f0-7d33-4930-a83a-8077a0465fda-kube-api-access-zxrkl\") pod \"maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49\" (UID: \"b7e097f0-7d33-4930-a83a-8077a0465fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49"
Jul 14 20:23:53.879135 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:23:53.878999    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7e097f0-7d33-4930-a83a-8077a0465fda-maas-api-tls\") pod \"maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49\" (UID: \"b7e097f0-7d33-4930-a83a-8077a0465fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49"
Jul 14 20:23:53.879723 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:23:53.879665    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e097f0-7d33-4930-a83a-8077a0465fda-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49\" (UID: \"b7e097f0-7d33-4930-a83a-8077a0465fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49"
Jul 14 20:23:53.881554 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:23:53.881533    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7e097f0-7d33-4930-a83a-8077a0465fda-maas-api-tls\") pod \"maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49\" (UID: \"b7e097f0-7d33-4930-a83a-8077a0465fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49"
Jul 14 20:23:53.888718 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:23:53.888694    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxrkl\" (UniqueName: \"kubernetes.io/projected/b7e097f0-7d33-4930-a83a-8077a0465fda-kube-api-access-zxrkl\") pod \"maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49\" (UID: \"b7e097f0-7d33-4930-a83a-8077a0465fda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49"
Jul 14 20:23:53.988432 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:23:53.988331    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49"
Jul 14 20:23:54.128734 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:23:54.128512    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49"]
Jul 14 20:23:54.131014 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:23:54.130983    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7e097f0_7d33_4930_a83a_8077a0465fda.slice/crio-ba704d8717e64020b9b788fb88d255d5457a1e1add94af4da336540fe67bf054 WatchSource:0}: Error finding container ba704d8717e64020b9b788fb88d255d5457a1e1add94af4da336540fe67bf054: Status 404 returned error can't find the container with id ba704d8717e64020b9b788fb88d255d5457a1e1add94af4da336540fe67bf054
Jul 14 20:23:54.248398 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:23:54.248304    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49" event={"ID":"b7e097f0-7d33-4930-a83a-8077a0465fda","Type":"ContainerStarted","Data":"ba704d8717e64020b9b788fb88d255d5457a1e1add94af4da336540fe67bf054"}
Jul 14 20:23:55.254195 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:23:55.254136    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49" event={"ID":"b7e097f0-7d33-4930-a83a-8077a0465fda","Type":"ContainerStarted","Data":"87995d698b2efa59122accd31ee57a3cacfe8a091ea7ae9585af11c0b24012ad"}
Jul 14 20:23:55.254609 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:23:55.254294    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49"
Jul 14 20:23:55.271696 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:23:55.271651    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49" podStartSLOduration=1.783587359 podStartE2EDuration="2.271636196s" totalImagesPullingTime="488.048837ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:23:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:23:55.270519432 +0000 UTC m=+2187.870823137" watchObservedRunningTime="2026-07-14 20:23:55.271636196 +0000 UTC m=+2187.871939898"
Jul 14 20:24:01.263892 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:24:01.263859    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49"
Jul 14 20:24:01.844895 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:24:01.844857    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq"]
Jul 14 20:24:01.849011 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:24:01.848987    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq"
Jul 14 20:24:01.851153 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:24:01.851130    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-eb3837-serving-cert\""
Jul 14 20:24:01.858148 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:24:01.858120    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq"]
Jul 14 20:24:01.955040 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:24:01.955001    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjnpm\" (UniqueName: \"kubernetes.io/projected/c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9-kube-api-access-mjnpm\") pod \"maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq\" (UID: \"c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq"
Jul 14 20:24:01.955278 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:24:01.955062    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq\" (UID: \"c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq"
Jul 14 20:24:01.955278 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:24:01.955142    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9-maas-api-tls\") pod \"maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq\" (UID: \"c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq"
Jul 14 20:24:02.056381 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:24:02.056341    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjnpm\" (UniqueName: \"kubernetes.io/projected/c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9-kube-api-access-mjnpm\") pod \"maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq\" (UID: \"c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq"
Jul 14 20:24:02.056575 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:24:02.056440    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq\" (UID: \"c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq"
Jul 14 20:24:02.056713 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:24:02.056690    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9-maas-api-tls\") pod \"maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq\" (UID: \"c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq"
Jul 14 20:24:02.057131 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:24:02.057111    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq\" (UID: \"c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq"
Jul 14 20:24:02.059184 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:24:02.059157    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9-maas-api-tls\") pod \"maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq\" (UID: \"c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq"
Jul 14 20:24:02.065374 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:24:02.065354    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjnpm\" (UniqueName: \"kubernetes.io/projected/c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9-kube-api-access-mjnpm\") pod \"maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq\" (UID: \"c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq"
Jul 14 20:24:02.162069 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:24:02.162034    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq"
Jul 14 20:24:02.311567 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:24:02.311396    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq"]
Jul 14 20:24:02.314495 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:24:02.314458    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc65aab96_7c3e_4be2_b9b7_40e6d4bcf4e9.slice/crio-69b3bc345001457c7c842c2ad9e975f829901445ea889411c63b3755c50e686a WatchSource:0}: Error finding container 69b3bc345001457c7c842c2ad9e975f829901445ea889411c63b3755c50e686a: Status 404 returned error can't find the container with id 69b3bc345001457c7c842c2ad9e975f829901445ea889411c63b3755c50e686a
Jul 14 20:24:03.295618 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:24:03.295582    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq" event={"ID":"c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9","Type":"ContainerStarted","Data":"0cb68bdb4d89fa054578e29eee96acd6a92b28955cb0d843cf6f14bd03519ca3"}
Jul 14 20:24:03.295787 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:24:03.295664    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq" event={"ID":"c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9","Type":"ContainerStarted","Data":"69b3bc345001457c7c842c2ad9e975f829901445ea889411c63b3755c50e686a"}
Jul 14 20:24:03.295787 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:24:03.295725    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq"
Jul 14 20:24:03.318339 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:24:03.318213    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq" podStartSLOduration=1.75011681 podStartE2EDuration="2.318194597s" totalImagesPullingTime="568.077787ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:24:03.310563812 +0000 UTC m=+2195.910867515" watchObservedRunningTime="2026-07-14 20:24:03.318194597 +0000 UTC m=+2195.918498296"
Jul 14 20:24:09.306111 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:24:09.306078    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq"
Jul 14 20:25:57.540025 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:25:57.539972    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49"]
Jul 14 20:25:57.540772 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:25:57.540255    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49" podUID="b7e097f0-7d33-4930-a83a-8077a0465fda" containerName="maas-api" containerID="cri-o://87995d698b2efa59122accd31ee57a3cacfe8a091ea7ae9585af11c0b24012ad" gracePeriod=30
Jul 14 20:25:57.790921 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:25:57.790857    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49"
Jul 14 20:25:57.860161 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:25:57.860123    2560 generic.go:356] "Generic (PLEG): container finished" podID="b7e097f0-7d33-4930-a83a-8077a0465fda" containerID="87995d698b2efa59122accd31ee57a3cacfe8a091ea7ae9585af11c0b24012ad" exitCode=0
Jul 14 20:25:57.860360 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:25:57.860200    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49"
Jul 14 20:25:57.860360 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:25:57.860215    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49" event={"ID":"b7e097f0-7d33-4930-a83a-8077a0465fda","Type":"ContainerDied","Data":"87995d698b2efa59122accd31ee57a3cacfe8a091ea7ae9585af11c0b24012ad"}
Jul 14 20:25:57.860360 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:25:57.860273    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49" event={"ID":"b7e097f0-7d33-4930-a83a-8077a0465fda","Type":"ContainerDied","Data":"ba704d8717e64020b9b788fb88d255d5457a1e1add94af4da336540fe67bf054"}
Jul 14 20:25:57.860360 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:25:57.860298    2560 scope.go:117] "RemoveContainer" containerID="87995d698b2efa59122accd31ee57a3cacfe8a091ea7ae9585af11c0b24012ad"
Jul 14 20:25:57.869242 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:25:57.869223    2560 scope.go:117] "RemoveContainer" containerID="87995d698b2efa59122accd31ee57a3cacfe8a091ea7ae9585af11c0b24012ad"
Jul 14 20:25:57.869509 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:25:57.869493    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87995d698b2efa59122accd31ee57a3cacfe8a091ea7ae9585af11c0b24012ad\": container with ID starting with 87995d698b2efa59122accd31ee57a3cacfe8a091ea7ae9585af11c0b24012ad not found: ID does not exist" containerID="87995d698b2efa59122accd31ee57a3cacfe8a091ea7ae9585af11c0b24012ad"
Jul 14 20:25:57.869552 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:25:57.869517    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87995d698b2efa59122accd31ee57a3cacfe8a091ea7ae9585af11c0b24012ad"} err="failed to get container status \"87995d698b2efa59122accd31ee57a3cacfe8a091ea7ae9585af11c0b24012ad\": rpc error: code = NotFound desc = could not find container \"87995d698b2efa59122accd31ee57a3cacfe8a091ea7ae9585af11c0b24012ad\": container with ID starting with 87995d698b2efa59122accd31ee57a3cacfe8a091ea7ae9585af11c0b24012ad not found: ID does not exist"
Jul 14 20:25:57.892829 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:25:57.892786    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7e097f0-7d33-4930-a83a-8077a0465fda-maas-api-tls\") pod \"b7e097f0-7d33-4930-a83a-8077a0465fda\" (UID: \"b7e097f0-7d33-4930-a83a-8077a0465fda\") "
Jul 14 20:25:57.893110 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:25:57.893088    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e097f0-7d33-4930-a83a-8077a0465fda-openshift-service-ca\") pod \"b7e097f0-7d33-4930-a83a-8077a0465fda\" (UID: \"b7e097f0-7d33-4930-a83a-8077a0465fda\") "
Jul 14 20:25:57.893203 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:25:57.893163    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxrkl\" (UniqueName: \"kubernetes.io/projected/b7e097f0-7d33-4930-a83a-8077a0465fda-kube-api-access-zxrkl\") pod \"b7e097f0-7d33-4930-a83a-8077a0465fda\" (UID: \"b7e097f0-7d33-4930-a83a-8077a0465fda\") "
Jul 14 20:25:57.893674 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:25:57.893651    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7e097f0-7d33-4930-a83a-8077a0465fda-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b7e097f0-7d33-4930-a83a-8077a0465fda" (UID: "b7e097f0-7d33-4930-a83a-8077a0465fda"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 20:25:57.895754 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:25:57.895732    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7e097f0-7d33-4930-a83a-8077a0465fda-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b7e097f0-7d33-4930-a83a-8077a0465fda" (UID: "b7e097f0-7d33-4930-a83a-8077a0465fda"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 20:25:57.895882 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:25:57.895862    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7e097f0-7d33-4930-a83a-8077a0465fda-kube-api-access-zxrkl" (OuterVolumeSpecName: "kube-api-access-zxrkl") pod "b7e097f0-7d33-4930-a83a-8077a0465fda" (UID: "b7e097f0-7d33-4930-a83a-8077a0465fda"). InnerVolumeSpecName "kube-api-access-zxrkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 20:25:57.994988 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:25:57.994954    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxrkl\" (UniqueName: \"kubernetes.io/projected/b7e097f0-7d33-4930-a83a-8077a0465fda-kube-api-access-zxrkl\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:25:57.994988 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:25:57.994982    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7e097f0-7d33-4930-a83a-8077a0465fda-maas-api-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:25:57.994988 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:25:57.994992    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e097f0-7d33-4930-a83a-8077a0465fda-openshift-service-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:25:58.176476 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:25:58.176399    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49"]
Jul 14 20:25:58.180626 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:25:58.180601    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d5be0-6cfc8c469b-z6f49"]
Jul 14 20:25:59.889494 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:25:59.889464    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7e097f0-7d33-4930-a83a-8077a0465fda" path="/var/lib/kubelet/pods/b7e097f0-7d33-4930-a83a-8077a0465fda/volumes"
Jul 14 20:26:23.757350 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:23.757230    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq"]
Jul 14 20:26:23.757834 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:23.757458    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq" podUID="c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9" containerName="maas-api" containerID="cri-o://0cb68bdb4d89fa054578e29eee96acd6a92b28955cb0d843cf6f14bd03519ca3" gracePeriod=30
Jul 14 20:26:23.990720 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:23.990683    2560 generic.go:356] "Generic (PLEG): container finished" podID="c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9" containerID="0cb68bdb4d89fa054578e29eee96acd6a92b28955cb0d843cf6f14bd03519ca3" exitCode=0
Jul 14 20:26:23.990860 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:23.990758    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq" event={"ID":"c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9","Type":"ContainerDied","Data":"0cb68bdb4d89fa054578e29eee96acd6a92b28955cb0d843cf6f14bd03519ca3"}
Jul 14 20:26:24.009395 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:24.009338    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq"
Jul 14 20:26:24.121930 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:24.121894    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9-openshift-service-ca\") pod \"c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9\" (UID: \"c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9\") "
Jul 14 20:26:24.121930 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:24.121934    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9-maas-api-tls\") pod \"c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9\" (UID: \"c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9\") "
Jul 14 20:26:24.122215 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:24.122003    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjnpm\" (UniqueName: \"kubernetes.io/projected/c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9-kube-api-access-mjnpm\") pod \"c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9\" (UID: \"c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9\") "
Jul 14 20:26:24.122371 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:24.122347    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9" (UID: "c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 20:26:24.124307 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:24.124277    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9" (UID: "c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 20:26:24.124418 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:24.124309    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9-kube-api-access-mjnpm" (OuterVolumeSpecName: "kube-api-access-mjnpm") pod "c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9" (UID: "c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9"). InnerVolumeSpecName "kube-api-access-mjnpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 20:26:24.222922 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:24.222888    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9-openshift-service-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:26:24.222922 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:24.222917    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9-maas-api-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:26:24.222922 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:24.222927    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjnpm\" (UniqueName: \"kubernetes.io/projected/c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9-kube-api-access-mjnpm\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:26:24.996602 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:24.996565    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq" event={"ID":"c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9","Type":"ContainerDied","Data":"69b3bc345001457c7c842c2ad9e975f829901445ea889411c63b3755c50e686a"}
Jul 14 20:26:24.997071 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:24.996623    2560 scope.go:117] "RemoveContainer" containerID="0cb68bdb4d89fa054578e29eee96acd6a92b28955cb0d843cf6f14bd03519ca3"
Jul 14 20:26:24.997071 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:24.996581    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq"
Jul 14 20:26:25.005068 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:26:25.005038    2560 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc65aab96_7c3e_4be2_b9b7_40e6d4bcf4e9.slice\": RecentStats: unable to find data in memory cache]"
Jul 14 20:26:25.018432 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:25.018408    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq"]
Jul 14 20:26:25.022188 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:25.022154    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3837-5cd4598487-8lbxq"]
Jul 14 20:26:25.890440 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:25.890397    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9" path="/var/lib/kubelet/pods/c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9/volumes"
Jul 14 20:26:43.612576 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:43.612543    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4"]
Jul 14 20:26:43.613132 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:43.613106    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9" containerName="maas-api"
Jul 14 20:26:43.613132 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:43.613121    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9" containerName="maas-api"
Jul 14 20:26:43.613276 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:43.613150    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7e097f0-7d33-4930-a83a-8077a0465fda" containerName="maas-api"
Jul 14 20:26:43.613276 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:43.613159    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7e097f0-7d33-4930-a83a-8077a0465fda" containerName="maas-api"
Jul 14 20:26:43.613342 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:43.613277    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c65aab96-7c3e-4be2-b9b7-40e6d4bcf4e9" containerName="maas-api"
Jul 14 20:26:43.613342 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:43.613293    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7e097f0-7d33-4930-a83a-8077a0465fda" containerName="maas-api"
Jul 14 20:26:43.616601 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:43.616581    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4"
Jul 14 20:26:43.619283 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:43.619258    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gfkms\""
Jul 14 20:26:43.619411 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:43.619253    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-409b4f-serving-cert\""
Jul 14 20:26:43.626032 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:43.625896    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4"]
Jul 14 20:26:43.698705 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:43.698673    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c7d3f3-188f-4365-ab1a-9bc5b604290d-openshift-service-ca\") pod \"maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4\" (UID: \"29c7d3f3-188f-4365-ab1a-9bc5b604290d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4"
Jul 14 20:26:43.698705 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:43.698710    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghms8\" (UniqueName: \"kubernetes.io/projected/29c7d3f3-188f-4365-ab1a-9bc5b604290d-kube-api-access-ghms8\") pod \"maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4\" (UID: \"29c7d3f3-188f-4365-ab1a-9bc5b604290d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4"
Jul 14 20:26:43.698938 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:43.698732    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c7d3f3-188f-4365-ab1a-9bc5b604290d-maas-api-tls\") pod \"maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4\" (UID: \"29c7d3f3-188f-4365-ab1a-9bc5b604290d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4"
Jul 14 20:26:43.799166 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:43.799131    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c7d3f3-188f-4365-ab1a-9bc5b604290d-maas-api-tls\") pod \"maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4\" (UID: \"29c7d3f3-188f-4365-ab1a-9bc5b604290d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4"
Jul 14 20:26:43.799347 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:43.799246    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c7d3f3-188f-4365-ab1a-9bc5b604290d-openshift-service-ca\") pod \"maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4\" (UID: \"29c7d3f3-188f-4365-ab1a-9bc5b604290d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4"
Jul 14 20:26:43.799347 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:43.799269    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghms8\" (UniqueName: \"kubernetes.io/projected/29c7d3f3-188f-4365-ab1a-9bc5b604290d-kube-api-access-ghms8\") pod \"maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4\" (UID: \"29c7d3f3-188f-4365-ab1a-9bc5b604290d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4"
Jul 14 20:26:43.799946 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:43.799871    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c7d3f3-188f-4365-ab1a-9bc5b604290d-openshift-service-ca\") pod \"maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4\" (UID: \"29c7d3f3-188f-4365-ab1a-9bc5b604290d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4"
Jul 14 20:26:43.801887 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:43.801861    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c7d3f3-188f-4365-ab1a-9bc5b604290d-maas-api-tls\") pod \"maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4\" (UID: \"29c7d3f3-188f-4365-ab1a-9bc5b604290d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4"
Jul 14 20:26:43.807369 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:43.807350    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghms8\" (UniqueName: \"kubernetes.io/projected/29c7d3f3-188f-4365-ab1a-9bc5b604290d-kube-api-access-ghms8\") pod \"maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4\" (UID: \"29c7d3f3-188f-4365-ab1a-9bc5b604290d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4"
Jul 14 20:26:43.929092 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:43.928994    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4"
Jul 14 20:26:44.073529 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:44.073502    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4"]
Jul 14 20:26:44.075493 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:26:44.075469    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c7d3f3_188f_4365_ab1a_9bc5b604290d.slice/crio-04dd2e74beae2f954edb47fb265ca4165ee98f4e9f5c651d9fc331251db09860 WatchSource:0}: Error finding container 04dd2e74beae2f954edb47fb265ca4165ee98f4e9f5c651d9fc331251db09860: Status 404 returned error can't find the container with id 04dd2e74beae2f954edb47fb265ca4165ee98f4e9f5c651d9fc331251db09860
Jul 14 20:26:44.076936 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:44.076917    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 20:26:44.091558 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:44.091529    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4" event={"ID":"29c7d3f3-188f-4365-ab1a-9bc5b604290d","Type":"ContainerStarted","Data":"04dd2e74beae2f954edb47fb265ca4165ee98f4e9f5c651d9fc331251db09860"}
Jul 14 20:26:45.098299 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:45.098202    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4" event={"ID":"29c7d3f3-188f-4365-ab1a-9bc5b604290d","Type":"ContainerStarted","Data":"de974e229671e6707590a472dfdf37ef0752bb02ef76e2ecf243b2eff71e3fdd"}
Jul 14 20:26:45.098299 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:45.098281    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4"
Jul 14 20:26:45.123894 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:45.123844    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4" podStartSLOduration=1.3671326910000001 podStartE2EDuration="2.123829062s" totalImagesPullingTime="756.696371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:26:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:26:45.121460113 +0000 UTC m=+2357.721763815" watchObservedRunningTime="2026-07-14 20:26:45.123829062 +0000 UTC m=+2357.724132820"
Jul 14 20:26:51.108662 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:51.108630    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4"
Jul 14 20:26:51.639811 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:51.639781    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl"]
Jul 14 20:26:51.643659 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:51.643639    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl"
Jul 14 20:26:51.645865 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:51.645844    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-0f00f6-serving-cert\""
Jul 14 20:26:51.654605 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:51.654583    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl"]
Jul 14 20:26:51.662472 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:51.662451    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe7bf93c-0423-4b65-a404-97ae7bb976cd-openshift-service-ca\") pod \"maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl\" (UID: \"fe7bf93c-0423-4b65-a404-97ae7bb976cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl"
Jul 14 20:26:51.662558 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:51.662493    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4whnl\" (UniqueName: \"kubernetes.io/projected/fe7bf93c-0423-4b65-a404-97ae7bb976cd-kube-api-access-4whnl\") pod \"maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl\" (UID: \"fe7bf93c-0423-4b65-a404-97ae7bb976cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl"
Jul 14 20:26:51.662558 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:51.662530    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe7bf93c-0423-4b65-a404-97ae7bb976cd-maas-api-tls\") pod \"maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl\" (UID: \"fe7bf93c-0423-4b65-a404-97ae7bb976cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl"
Jul 14 20:26:51.763591 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:51.763550    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe7bf93c-0423-4b65-a404-97ae7bb976cd-openshift-service-ca\") pod \"maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl\" (UID: \"fe7bf93c-0423-4b65-a404-97ae7bb976cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl"
Jul 14 20:26:51.763806 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:51.763606    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4whnl\" (UniqueName: \"kubernetes.io/projected/fe7bf93c-0423-4b65-a404-97ae7bb976cd-kube-api-access-4whnl\") pod \"maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl\" (UID: \"fe7bf93c-0423-4b65-a404-97ae7bb976cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl"
Jul 14 20:26:51.763806 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:51.763630    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe7bf93c-0423-4b65-a404-97ae7bb976cd-maas-api-tls\") pod \"maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl\" (UID: \"fe7bf93c-0423-4b65-a404-97ae7bb976cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl"
Jul 14 20:26:51.763806 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:26:51.763759    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-serving-cert: secret "maas-api-e2e-api-b-0f00f6-serving-cert" not found
Jul 14 20:26:51.763997 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:26:51.763831    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe7bf93c-0423-4b65-a404-97ae7bb976cd-maas-api-tls podName:fe7bf93c-0423-4b65-a404-97ae7bb976cd nodeName:}" failed. No retries permitted until 2026-07-14 20:26:52.26381478 +0000 UTC m=+2364.864118461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fe7bf93c-0423-4b65-a404-97ae7bb976cd-maas-api-tls") pod "maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl" (UID: "fe7bf93c-0423-4b65-a404-97ae7bb976cd") : secret "maas-api-e2e-api-b-0f00f6-serving-cert" not found
Jul 14 20:26:51.764346 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:51.764322    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe7bf93c-0423-4b65-a404-97ae7bb976cd-openshift-service-ca\") pod \"maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl\" (UID: \"fe7bf93c-0423-4b65-a404-97ae7bb976cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl"
Jul 14 20:26:51.780865 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:51.780839    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4whnl\" (UniqueName: \"kubernetes.io/projected/fe7bf93c-0423-4b65-a404-97ae7bb976cd-kube-api-access-4whnl\") pod \"maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl\" (UID: \"fe7bf93c-0423-4b65-a404-97ae7bb976cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl"
Jul 14 20:26:52.268435 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:52.268393    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe7bf93c-0423-4b65-a404-97ae7bb976cd-maas-api-tls\") pod \"maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl\" (UID: \"fe7bf93c-0423-4b65-a404-97ae7bb976cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl"
Jul 14 20:26:52.270921 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:52.270898    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe7bf93c-0423-4b65-a404-97ae7bb976cd-maas-api-tls\") pod \"maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl\" (UID: \"fe7bf93c-0423-4b65-a404-97ae7bb976cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl"
Jul 14 20:26:52.555160 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:52.555122    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl"
Jul 14 20:26:52.693770 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:52.693736    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl"]
Jul 14 20:26:52.695321 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:26:52.695292    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe7bf93c_0423_4b65_a404_97ae7bb976cd.slice/crio-0aaa055e7ebe1fe224e8b2395594b0792f6177868e0f2a1a7ec57136704a8c8e WatchSource:0}: Error finding container 0aaa055e7ebe1fe224e8b2395594b0792f6177868e0f2a1a7ec57136704a8c8e: Status 404 returned error can't find the container with id 0aaa055e7ebe1fe224e8b2395594b0792f6177868e0f2a1a7ec57136704a8c8e
Jul 14 20:26:53.137805 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:53.137769    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl" event={"ID":"fe7bf93c-0423-4b65-a404-97ae7bb976cd","Type":"ContainerStarted","Data":"0aaa055e7ebe1fe224e8b2395594b0792f6177868e0f2a1a7ec57136704a8c8e"}
Jul 14 20:26:54.144348 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:54.144314    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl" event={"ID":"fe7bf93c-0423-4b65-a404-97ae7bb976cd","Type":"ContainerStarted","Data":"42071fd70bf1c3a37774c6071326a0b7dc47e4ddcdf2fcc882eae570cb41586f"}
Jul 14 20:26:54.144794 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:54.144482    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl"
Jul 14 20:26:54.161977 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:26:54.161931    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl" podStartSLOduration=2.675406434 podStartE2EDuration="3.1619172s" totalImagesPullingTime="486.510766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:26:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:26:54.160255742 +0000 UTC m=+2366.760559436" watchObservedRunningTime="2026-07-14 20:26:54.1619172 +0000 UTC m=+2366.762220904"
Jul 14 20:27:00.155370 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:00.155340    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl"
Jul 14 20:27:15.247726 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:15.247688    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4"]
Jul 14 20:27:15.248370 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:15.247945    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4" podUID="29c7d3f3-188f-4365-ab1a-9bc5b604290d" containerName="maas-api" containerID="cri-o://de974e229671e6707590a472dfdf37ef0752bb02ef76e2ecf243b2eff71e3fdd" gracePeriod=30
Jul 14 20:27:15.495345 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:15.495317    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4"
Jul 14 20:27:15.590530 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:15.590491    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c7d3f3-188f-4365-ab1a-9bc5b604290d-openshift-service-ca\") pod \"29c7d3f3-188f-4365-ab1a-9bc5b604290d\" (UID: \"29c7d3f3-188f-4365-ab1a-9bc5b604290d\") "
Jul 14 20:27:15.590725 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:15.590569    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghms8\" (UniqueName: \"kubernetes.io/projected/29c7d3f3-188f-4365-ab1a-9bc5b604290d-kube-api-access-ghms8\") pod \"29c7d3f3-188f-4365-ab1a-9bc5b604290d\" (UID: \"29c7d3f3-188f-4365-ab1a-9bc5b604290d\") "
Jul 14 20:27:15.590725 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:15.590597    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c7d3f3-188f-4365-ab1a-9bc5b604290d-maas-api-tls\") pod \"29c7d3f3-188f-4365-ab1a-9bc5b604290d\" (UID: \"29c7d3f3-188f-4365-ab1a-9bc5b604290d\") "
Jul 14 20:27:15.590975 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:15.590947    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c7d3f3-188f-4365-ab1a-9bc5b604290d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "29c7d3f3-188f-4365-ab1a-9bc5b604290d" (UID: "29c7d3f3-188f-4365-ab1a-9bc5b604290d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 20:27:15.592932 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:15.592901    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c7d3f3-188f-4365-ab1a-9bc5b604290d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "29c7d3f3-188f-4365-ab1a-9bc5b604290d" (UID: "29c7d3f3-188f-4365-ab1a-9bc5b604290d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 20:27:15.592932 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:15.592913    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c7d3f3-188f-4365-ab1a-9bc5b604290d-kube-api-access-ghms8" (OuterVolumeSpecName: "kube-api-access-ghms8") pod "29c7d3f3-188f-4365-ab1a-9bc5b604290d" (UID: "29c7d3f3-188f-4365-ab1a-9bc5b604290d"). InnerVolumeSpecName "kube-api-access-ghms8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 20:27:15.691968 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:15.691934    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghms8\" (UniqueName: \"kubernetes.io/projected/29c7d3f3-188f-4365-ab1a-9bc5b604290d-kube-api-access-ghms8\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:27:15.691968 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:15.691963    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c7d3f3-188f-4365-ab1a-9bc5b604290d-maas-api-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:27:15.691968 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:15.691973    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c7d3f3-188f-4365-ab1a-9bc5b604290d-openshift-service-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:27:16.259214 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:16.259099    2560 generic.go:356] "Generic (PLEG): container finished" podID="29c7d3f3-188f-4365-ab1a-9bc5b604290d" containerID="de974e229671e6707590a472dfdf37ef0752bb02ef76e2ecf243b2eff71e3fdd" exitCode=0
Jul 14 20:27:16.259214 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:16.259150    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4"
Jul 14 20:27:16.259727 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:16.259215    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4" event={"ID":"29c7d3f3-188f-4365-ab1a-9bc5b604290d","Type":"ContainerDied","Data":"de974e229671e6707590a472dfdf37ef0752bb02ef76e2ecf243b2eff71e3fdd"}
Jul 14 20:27:16.259727 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:16.259273    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4" event={"ID":"29c7d3f3-188f-4365-ab1a-9bc5b604290d","Type":"ContainerDied","Data":"04dd2e74beae2f954edb47fb265ca4165ee98f4e9f5c651d9fc331251db09860"}
Jul 14 20:27:16.259727 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:16.259294    2560 scope.go:117] "RemoveContainer" containerID="de974e229671e6707590a472dfdf37ef0752bb02ef76e2ecf243b2eff71e3fdd"
Jul 14 20:27:16.268461 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:16.268437    2560 scope.go:117] "RemoveContainer" containerID="de974e229671e6707590a472dfdf37ef0752bb02ef76e2ecf243b2eff71e3fdd"
Jul 14 20:27:16.268714 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:27:16.268695    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de974e229671e6707590a472dfdf37ef0752bb02ef76e2ecf243b2eff71e3fdd\": container with ID starting with de974e229671e6707590a472dfdf37ef0752bb02ef76e2ecf243b2eff71e3fdd not found: ID does not exist" containerID="de974e229671e6707590a472dfdf37ef0752bb02ef76e2ecf243b2eff71e3fdd"
Jul 14 20:27:16.268777 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:16.268723    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de974e229671e6707590a472dfdf37ef0752bb02ef76e2ecf243b2eff71e3fdd"} err="failed to get container status \"de974e229671e6707590a472dfdf37ef0752bb02ef76e2ecf243b2eff71e3fdd\": rpc error: code = NotFound desc = could not find container \"de974e229671e6707590a472dfdf37ef0752bb02ef76e2ecf243b2eff71e3fdd\": container with ID starting with de974e229671e6707590a472dfdf37ef0752bb02ef76e2ecf243b2eff71e3fdd not found: ID does not exist"
Jul 14 20:27:16.278321 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:16.278299    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4"]
Jul 14 20:27:16.282209 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:16.282167    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-409b4f-7bc4f67bb6-42pg4"]
Jul 14 20:27:17.889289 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:17.889259    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29c7d3f3-188f-4365-ab1a-9bc5b604290d" path="/var/lib/kubelet/pods/29c7d3f3-188f-4365-ab1a-9bc5b604290d/volumes"
Jul 14 20:27:28.014065 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:28.014034    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tzr6_4a36af4c-9eac-465f-9a0e-d65a382530f7/ovn-acl-logging/0.log"
Jul 14 20:27:28.027767 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:28.027744    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tzr6_4a36af4c-9eac-465f-9a0e-d65a382530f7/ovn-acl-logging/0.log"
Jul 14 20:27:30.561697 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:30.561662    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0f00f6-p2msv"]
Jul 14 20:27:30.562087 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:30.562045    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29c7d3f3-188f-4365-ab1a-9bc5b604290d" containerName="maas-api"
Jul 14 20:27:30.562087 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:30.562055    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="29c7d3f3-188f-4365-ab1a-9bc5b604290d" containerName="maas-api"
Jul 14 20:27:30.562161 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:30.562113    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="29c7d3f3-188f-4365-ab1a-9bc5b604290d" containerName="maas-api"
Jul 14 20:27:30.566606 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:30.566589    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0f00f6-p2msv"
Jul 14 20:27:30.569262 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:30.569053    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-56w6s\""
Jul 14 20:27:30.572338 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:30.572312    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0f00f6-p2msv"]
Jul 14 20:27:30.628012 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:30.627963    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/056d89e4-a506-47a8-bd63-43d4cec3c17a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0f00f6-p2msv\" (UID: \"056d89e4-a506-47a8-bd63-43d4cec3c17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0f00f6-p2msv"
Jul 14 20:27:30.729262 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:30.729224    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/056d89e4-a506-47a8-bd63-43d4cec3c17a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0f00f6-p2msv\" (UID: \"056d89e4-a506-47a8-bd63-43d4cec3c17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0f00f6-p2msv"
Jul 14 20:27:30.729889 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:30.729868    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/056d89e4-a506-47a8-bd63-43d4cec3c17a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0f00f6-p2msv\" (UID: \"056d89e4-a506-47a8-bd63-43d4cec3c17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0f00f6-p2msv"
Jul 14 20:27:30.877868 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:30.877783    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0f00f6-p2msv"
Jul 14 20:27:31.015913 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:31.015887    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0f00f6-p2msv"]
Jul 14 20:27:31.017427 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:27:31.017394    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056d89e4_a506_47a8_bd63_43d4cec3c17a.slice/crio-bcccc4547724096db5c6446e71c92bac612c401123e558b48a9d4da5a246e6e7 WatchSource:0}: Error finding container bcccc4547724096db5c6446e71c92bac612c401123e558b48a9d4da5a246e6e7: Status 404 returned error can't find the container with id bcccc4547724096db5c6446e71c92bac612c401123e558b48a9d4da5a246e6e7
Jul 14 20:27:31.338664 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:31.338629    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0f00f6-p2msv" event={"ID":"056d89e4-a506-47a8-bd63-43d4cec3c17a","Type":"ContainerStarted","Data":"ad0dc32b6333bf165db9baeb8a8dac5d795eff590e0232045344877b73e55e0b"}
Jul 14 20:27:31.338853 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:31.338696    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0f00f6-p2msv" event={"ID":"056d89e4-a506-47a8-bd63-43d4cec3c17a","Type":"ContainerStarted","Data":"bcccc4547724096db5c6446e71c92bac612c401123e558b48a9d4da5a246e6e7"}
Jul 14 20:27:31.354658 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:31.354605    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0f00f6-p2msv" podStartSLOduration=1.35458934 podStartE2EDuration="1.35458934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:27:31.353394403 +0000 UTC m=+2403.953698107" watchObservedRunningTime="2026-07-14 20:27:31.35458934 +0000 UTC m=+2403.954893042"
Jul 14 20:27:32.350319 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:32.350286    2560 generic.go:356] "Generic (PLEG): container finished" podID="056d89e4-a506-47a8-bd63-43d4cec3c17a" containerID="ad0dc32b6333bf165db9baeb8a8dac5d795eff590e0232045344877b73e55e0b" exitCode=0
Jul 14 20:27:32.350319 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:32.350323    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0f00f6-p2msv" event={"ID":"056d89e4-a506-47a8-bd63-43d4cec3c17a","Type":"ContainerDied","Data":"ad0dc32b6333bf165db9baeb8a8dac5d795eff590e0232045344877b73e55e0b"}
Jul 14 20:27:33.484564 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:33.484541    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0f00f6-p2msv"
Jul 14 20:27:33.558132 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:33.558102    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/056d89e4-a506-47a8-bd63-43d4cec3c17a-maas-api-service-ca\") pod \"056d89e4-a506-47a8-bd63-43d4cec3c17a\" (UID: \"056d89e4-a506-47a8-bd63-43d4cec3c17a\") "
Jul 14 20:27:33.558485 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:33.558460    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056d89e4-a506-47a8-bd63-43d4cec3c17a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "056d89e4-a506-47a8-bd63-43d4cec3c17a" (UID: "056d89e4-a506-47a8-bd63-43d4cec3c17a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 20:27:33.659216 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:33.659106    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/056d89e4-a506-47a8-bd63-43d4cec3c17a-maas-api-service-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:27:34.361726 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:34.361692    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0f00f6-p2msv" event={"ID":"056d89e4-a506-47a8-bd63-43d4cec3c17a","Type":"ContainerDied","Data":"bcccc4547724096db5c6446e71c92bac612c401123e558b48a9d4da5a246e6e7"}
Jul 14 20:27:34.361726 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:34.361716    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0f00f6-p2msv"
Jul 14 20:27:34.361950 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:34.361741    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcccc4547724096db5c6446e71c92bac612c401123e558b48a9d4da5a246e6e7"
Jul 14 20:27:40.998749 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:40.998717    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl"]
Jul 14 20:27:40.999199 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:40.998963    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl" podUID="fe7bf93c-0423-4b65-a404-97ae7bb976cd" containerName="maas-api" containerID="cri-o://42071fd70bf1c3a37774c6071326a0b7dc47e4ddcdf2fcc882eae570cb41586f" gracePeriod=30
Jul 14 20:27:41.241906 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:41.241881    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl"
Jul 14 20:27:41.327700 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:41.327662    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4whnl\" (UniqueName: \"kubernetes.io/projected/fe7bf93c-0423-4b65-a404-97ae7bb976cd-kube-api-access-4whnl\") pod \"fe7bf93c-0423-4b65-a404-97ae7bb976cd\" (UID: \"fe7bf93c-0423-4b65-a404-97ae7bb976cd\") "
Jul 14 20:27:41.327871 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:41.327717    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe7bf93c-0423-4b65-a404-97ae7bb976cd-openshift-service-ca\") pod \"fe7bf93c-0423-4b65-a404-97ae7bb976cd\" (UID: \"fe7bf93c-0423-4b65-a404-97ae7bb976cd\") "
Jul 14 20:27:41.327871 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:41.327810    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe7bf93c-0423-4b65-a404-97ae7bb976cd-maas-api-tls\") pod \"fe7bf93c-0423-4b65-a404-97ae7bb976cd\" (UID: \"fe7bf93c-0423-4b65-a404-97ae7bb976cd\") "
Jul 14 20:27:41.328106 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:41.328080    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe7bf93c-0423-4b65-a404-97ae7bb976cd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fe7bf93c-0423-4b65-a404-97ae7bb976cd" (UID: "fe7bf93c-0423-4b65-a404-97ae7bb976cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 20:27:41.330057 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:41.330027    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe7bf93c-0423-4b65-a404-97ae7bb976cd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fe7bf93c-0423-4b65-a404-97ae7bb976cd" (UID: "fe7bf93c-0423-4b65-a404-97ae7bb976cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 20:27:41.330057 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:41.330033    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe7bf93c-0423-4b65-a404-97ae7bb976cd-kube-api-access-4whnl" (OuterVolumeSpecName: "kube-api-access-4whnl") pod "fe7bf93c-0423-4b65-a404-97ae7bb976cd" (UID: "fe7bf93c-0423-4b65-a404-97ae7bb976cd"). InnerVolumeSpecName "kube-api-access-4whnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 20:27:41.395333 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:41.395299    2560 generic.go:356] "Generic (PLEG): container finished" podID="fe7bf93c-0423-4b65-a404-97ae7bb976cd" containerID="42071fd70bf1c3a37774c6071326a0b7dc47e4ddcdf2fcc882eae570cb41586f" exitCode=0
Jul 14 20:27:41.395509 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:41.395346    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl"
Jul 14 20:27:41.395509 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:41.395393    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl" event={"ID":"fe7bf93c-0423-4b65-a404-97ae7bb976cd","Type":"ContainerDied","Data":"42071fd70bf1c3a37774c6071326a0b7dc47e4ddcdf2fcc882eae570cb41586f"}
Jul 14 20:27:41.395509 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:41.395454    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl" event={"ID":"fe7bf93c-0423-4b65-a404-97ae7bb976cd","Type":"ContainerDied","Data":"0aaa055e7ebe1fe224e8b2395594b0792f6177868e0f2a1a7ec57136704a8c8e"}
Jul 14 20:27:41.395509 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:41.395479    2560 scope.go:117] "RemoveContainer" containerID="42071fd70bf1c3a37774c6071326a0b7dc47e4ddcdf2fcc882eae570cb41586f"
Jul 14 20:27:41.405093 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:41.405075    2560 scope.go:117] "RemoveContainer" containerID="42071fd70bf1c3a37774c6071326a0b7dc47e4ddcdf2fcc882eae570cb41586f"
Jul 14 20:27:41.405356 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:27:41.405337    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42071fd70bf1c3a37774c6071326a0b7dc47e4ddcdf2fcc882eae570cb41586f\": container with ID starting with 42071fd70bf1c3a37774c6071326a0b7dc47e4ddcdf2fcc882eae570cb41586f not found: ID does not exist" containerID="42071fd70bf1c3a37774c6071326a0b7dc47e4ddcdf2fcc882eae570cb41586f"
Jul 14 20:27:41.405428 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:41.405367    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42071fd70bf1c3a37774c6071326a0b7dc47e4ddcdf2fcc882eae570cb41586f"} err="failed to get container status \"42071fd70bf1c3a37774c6071326a0b7dc47e4ddcdf2fcc882eae570cb41586f\": rpc error: code = NotFound desc = could not find container \"42071fd70bf1c3a37774c6071326a0b7dc47e4ddcdf2fcc882eae570cb41586f\": container with ID starting with 42071fd70bf1c3a37774c6071326a0b7dc47e4ddcdf2fcc882eae570cb41586f not found: ID does not exist"
Jul 14 20:27:41.416184 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:41.416144    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl"]
Jul 14 20:27:41.420463 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:41.420443    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0f00f6-858dbcb9c8-sz5rl"]
Jul 14 20:27:41.428506 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:41.428486    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe7bf93c-0423-4b65-a404-97ae7bb976cd-openshift-service-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:27:41.428589 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:41.428509    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe7bf93c-0423-4b65-a404-97ae7bb976cd-maas-api-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:27:41.428589 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:41.428524    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4whnl\" (UniqueName: \"kubernetes.io/projected/fe7bf93c-0423-4b65-a404-97ae7bb976cd-kube-api-access-4whnl\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:27:41.890226 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:27:41.890167    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe7bf93c-0423-4b65-a404-97ae7bb976cd" path="/var/lib/kubelet/pods/fe7bf93c-0423-4b65-a404-97ae7bb976cd/volumes"
Jul 14 20:28:00.049102 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:00.049016    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj"]
Jul 14 20:28:00.049545 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:00.049493    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="056d89e4-a506-47a8-bd63-43d4cec3c17a" containerName="revoke-keys"
Jul 14 20:28:00.049545 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:00.049509    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="056d89e4-a506-47a8-bd63-43d4cec3c17a" containerName="revoke-keys"
Jul 14 20:28:00.049545 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:00.049520    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe7bf93c-0423-4b65-a404-97ae7bb976cd" containerName="maas-api"
Jul 14 20:28:00.049545 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:00.049525    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe7bf93c-0423-4b65-a404-97ae7bb976cd" containerName="maas-api"
Jul 14 20:28:00.049676 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:00.049596    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe7bf93c-0423-4b65-a404-97ae7bb976cd" containerName="maas-api"
Jul 14 20:28:00.049676 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:00.049607    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="056d89e4-a506-47a8-bd63-43d4cec3c17a" containerName="revoke-keys"
Jul 14 20:28:00.054057 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:00.054039    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj"
Jul 14 20:28:00.056424 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:00.056390    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-10ef4d-serving-cert\""
Jul 14 20:28:00.056424 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:00.056394    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gfkms\""
Jul 14 20:28:00.063721 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:00.063695    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj"]
Jul 14 20:28:00.196546 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:00.196516    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knwqr\" (UniqueName: \"kubernetes.io/projected/7705d7ef-f85a-4ada-b3d1-842e234be770-kube-api-access-knwqr\") pod \"maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj\" (UID: \"7705d7ef-f85a-4ada-b3d1-842e234be770\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj"
Jul 14 20:28:00.196746 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:00.196570    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7705d7ef-f85a-4ada-b3d1-842e234be770-openshift-service-ca\") pod \"maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj\" (UID: \"7705d7ef-f85a-4ada-b3d1-842e234be770\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj"
Jul 14 20:28:00.196746 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:00.196601    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7705d7ef-f85a-4ada-b3d1-842e234be770-maas-api-tls\") pod \"maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj\" (UID: \"7705d7ef-f85a-4ada-b3d1-842e234be770\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj"
Jul 14 20:28:00.297794 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:00.297754    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knwqr\" (UniqueName: \"kubernetes.io/projected/7705d7ef-f85a-4ada-b3d1-842e234be770-kube-api-access-knwqr\") pod \"maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj\" (UID: \"7705d7ef-f85a-4ada-b3d1-842e234be770\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj"
Jul 14 20:28:00.298009 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:00.297806    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7705d7ef-f85a-4ada-b3d1-842e234be770-openshift-service-ca\") pod \"maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj\" (UID: \"7705d7ef-f85a-4ada-b3d1-842e234be770\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj"
Jul 14 20:28:00.298009 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:00.297834    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7705d7ef-f85a-4ada-b3d1-842e234be770-maas-api-tls\") pod \"maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj\" (UID: \"7705d7ef-f85a-4ada-b3d1-842e234be770\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj"
Jul 14 20:28:00.298009 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:28:00.297967    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-serving-cert: secret "maas-api-e2e-shared-a-10ef4d-serving-cert" not found
Jul 14 20:28:00.298220 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:28:00.298044    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7705d7ef-f85a-4ada-b3d1-842e234be770-maas-api-tls podName:7705d7ef-f85a-4ada-b3d1-842e234be770 nodeName:}" failed. No retries permitted until 2026-07-14 20:28:00.798023599 +0000 UTC m=+2433.398327280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7705d7ef-f85a-4ada-b3d1-842e234be770-maas-api-tls") pod "maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj" (UID: "7705d7ef-f85a-4ada-b3d1-842e234be770") : secret "maas-api-e2e-shared-a-10ef4d-serving-cert" not found
Jul 14 20:28:00.298529 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:00.298508    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7705d7ef-f85a-4ada-b3d1-842e234be770-openshift-service-ca\") pod \"maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj\" (UID: \"7705d7ef-f85a-4ada-b3d1-842e234be770\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj"
Jul 14 20:28:00.306143 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:00.306087    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knwqr\" (UniqueName: \"kubernetes.io/projected/7705d7ef-f85a-4ada-b3d1-842e234be770-kube-api-access-knwqr\") pod \"maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj\" (UID: \"7705d7ef-f85a-4ada-b3d1-842e234be770\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj"
Jul 14 20:28:00.802781 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:00.802749    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7705d7ef-f85a-4ada-b3d1-842e234be770-maas-api-tls\") pod \"maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj\" (UID: \"7705d7ef-f85a-4ada-b3d1-842e234be770\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj"
Jul 14 20:28:00.805474 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:00.805450    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7705d7ef-f85a-4ada-b3d1-842e234be770-maas-api-tls\") pod \"maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj\" (UID: \"7705d7ef-f85a-4ada-b3d1-842e234be770\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj"
Jul 14 20:28:00.965829 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:00.965790    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj"
Jul 14 20:28:01.106098 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:01.106011    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj"]
Jul 14 20:28:01.109564 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:28:01.109541    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7705d7ef_f85a_4ada_b3d1_842e234be770.slice/crio-5d8e216898be2e6dfc4e7b19c270ed4102d6bf0d95bcbdd06ceabbcee97d2c2b WatchSource:0}: Error finding container 5d8e216898be2e6dfc4e7b19c270ed4102d6bf0d95bcbdd06ceabbcee97d2c2b: Status 404 returned error can't find the container with id 5d8e216898be2e6dfc4e7b19c270ed4102d6bf0d95bcbdd06ceabbcee97d2c2b
Jul 14 20:28:01.494565 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:01.494478    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj" event={"ID":"7705d7ef-f85a-4ada-b3d1-842e234be770","Type":"ContainerStarted","Data":"5d8e216898be2e6dfc4e7b19c270ed4102d6bf0d95bcbdd06ceabbcee97d2c2b"}
Jul 14 20:28:02.500919 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:02.500879    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj" event={"ID":"7705d7ef-f85a-4ada-b3d1-842e234be770","Type":"ContainerStarted","Data":"591b1bc8647c4e2be7f6b5cad163aa856054f7985734ddbac7cdba9f99a8e415"}
Jul 14 20:28:02.501403 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:02.500946    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj"
Jul 14 20:28:02.519071 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:02.519011    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj" podStartSLOduration=2.036884243 podStartE2EDuration="2.518991261s" totalImagesPullingTime="482.107018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:28:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:28:02.517280608 +0000 UTC m=+2435.117584313" watchObservedRunningTime="2026-07-14 20:28:02.518991261 +0000 UTC m=+2435.119294963"
Jul 14 20:28:08.124729 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:08.124695    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz"]
Jul 14 20:28:08.128668 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:08.128646    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz"
Jul 14 20:28:08.130892 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:08.130870    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-d04b66-serving-cert\""
Jul 14 20:28:08.136613 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:08.136381    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz"]
Jul 14 20:28:08.271402 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:08.271366    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/433df555-f927-4267-aaeb-fb74aaa17f8a-maas-api-tls\") pod \"maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz\" (UID: \"433df555-f927-4267-aaeb-fb74aaa17f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz"
Jul 14 20:28:08.271570 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:08.271446    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/433df555-f927-4267-aaeb-fb74aaa17f8a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz\" (UID: \"433df555-f927-4267-aaeb-fb74aaa17f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz"
Jul 14 20:28:08.271570 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:08.271481    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pfvq\" (UniqueName: \"kubernetes.io/projected/433df555-f927-4267-aaeb-fb74aaa17f8a-kube-api-access-8pfvq\") pod \"maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz\" (UID: \"433df555-f927-4267-aaeb-fb74aaa17f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz"
Jul 14 20:28:08.372824 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:08.372780    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/433df555-f927-4267-aaeb-fb74aaa17f8a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz\" (UID: \"433df555-f927-4267-aaeb-fb74aaa17f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz"
Jul 14 20:28:08.373003 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:08.372838    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pfvq\" (UniqueName: \"kubernetes.io/projected/433df555-f927-4267-aaeb-fb74aaa17f8a-kube-api-access-8pfvq\") pod \"maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz\" (UID: \"433df555-f927-4267-aaeb-fb74aaa17f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz"
Jul 14 20:28:08.373003 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:08.372950    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/433df555-f927-4267-aaeb-fb74aaa17f8a-maas-api-tls\") pod \"maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz\" (UID: \"433df555-f927-4267-aaeb-fb74aaa17f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz"
Jul 14 20:28:08.373119 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:28:08.373102    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-serving-cert: secret "maas-api-e2e-shared-b-d04b66-serving-cert" not found
Jul 14 20:28:08.373234 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:28:08.373209    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/433df555-f927-4267-aaeb-fb74aaa17f8a-maas-api-tls podName:433df555-f927-4267-aaeb-fb74aaa17f8a nodeName:}" failed. No retries permitted until 2026-07-14 20:28:08.873153951 +0000 UTC m=+2441.473457634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/433df555-f927-4267-aaeb-fb74aaa17f8a-maas-api-tls") pod "maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz" (UID: "433df555-f927-4267-aaeb-fb74aaa17f8a") : secret "maas-api-e2e-shared-b-d04b66-serving-cert" not found
Jul 14 20:28:08.373530 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:08.373513    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/433df555-f927-4267-aaeb-fb74aaa17f8a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz\" (UID: \"433df555-f927-4267-aaeb-fb74aaa17f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz"
Jul 14 20:28:08.383703 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:08.383642    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pfvq\" (UniqueName: \"kubernetes.io/projected/433df555-f927-4267-aaeb-fb74aaa17f8a-kube-api-access-8pfvq\") pod \"maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz\" (UID: \"433df555-f927-4267-aaeb-fb74aaa17f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz"
Jul 14 20:28:08.511932 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:08.511904    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj"
Jul 14 20:28:08.877469 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:08.877431    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/433df555-f927-4267-aaeb-fb74aaa17f8a-maas-api-tls\") pod \"maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz\" (UID: \"433df555-f927-4267-aaeb-fb74aaa17f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz"
Jul 14 20:28:08.880278 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:08.880252    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/433df555-f927-4267-aaeb-fb74aaa17f8a-maas-api-tls\") pod \"maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz\" (UID: \"433df555-f927-4267-aaeb-fb74aaa17f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz"
Jul 14 20:28:09.040314 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:09.040274    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz"
Jul 14 20:28:09.179301 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:09.179270    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz"]
Jul 14 20:28:09.181383 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:28:09.181358    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433df555_f927_4267_aaeb_fb74aaa17f8a.slice/crio-356a52dfdb10f5c74604202fad23c2dbdfaa8b15d749d03f43324df78c49d552 WatchSource:0}: Error finding container 356a52dfdb10f5c74604202fad23c2dbdfaa8b15d749d03f43324df78c49d552: Status 404 returned error can't find the container with id 356a52dfdb10f5c74604202fad23c2dbdfaa8b15d749d03f43324df78c49d552
Jul 14 20:28:09.538695 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:09.538665    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz" event={"ID":"433df555-f927-4267-aaeb-fb74aaa17f8a","Type":"ContainerStarted","Data":"356a52dfdb10f5c74604202fad23c2dbdfaa8b15d749d03f43324df78c49d552"}
Jul 14 20:28:10.545636 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:10.545600    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz" event={"ID":"433df555-f927-4267-aaeb-fb74aaa17f8a","Type":"ContainerStarted","Data":"3dfb27989cd828c746d300bd3ee8eff7564a6087ce2518cb7f61f91ad6487c91"}
Jul 14 20:28:10.546064 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:10.545685    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz"
Jul 14 20:28:10.569228 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:10.569159    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz" podStartSLOduration=2.258465638 podStartE2EDuration="2.569142523s" totalImagesPullingTime="310.676885ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:28:10.566545459 +0000 UTC m=+2443.166849177" watchObservedRunningTime="2026-07-14 20:28:10.569142523 +0000 UTC m=+2443.169446226"
Jul 14 20:28:16.556521 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:16.556493    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz"
Jul 14 20:28:38.244739 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:38.244318    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"]
Jul 14 20:28:38.248137 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:38.248120    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"
Jul 14 20:28:38.251858 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:38.251829    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d04b66\"/\"openshift-service-ca.crt\""
Jul 14 20:28:38.252018 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:38.251904    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d04b66\"/\"kube-root-ca.crt\""
Jul 14 20:28:38.252018 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:38.251928    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d04b66\"/\"auth-test-model-d04b66-kserve-self-signed-certs\""
Jul 14 20:28:38.252018 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:38.251837    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d04b66\"/\"default-dockercfg-ndhvd\""
Jul 14 20:28:38.257937 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:38.257915    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"]
Jul 14 20:28:38.359249 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:38.359204    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-dshm\") pod \"auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw\" (UID: \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\") " pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"
Jul 14 20:28:38.359249 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:38.359253    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-tmp-dir\") pod \"auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw\" (UID: \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\") " pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"
Jul 14 20:28:38.359510 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:38.359285    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj8xw\" (UniqueName: \"kubernetes.io/projected/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-kube-api-access-hj8xw\") pod \"auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw\" (UID: \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\") " pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"
Jul 14 20:28:38.359510 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:38.359333    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-home\") pod \"auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw\" (UID: \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\") " pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"
Jul 14 20:28:38.359510 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:38.359347    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-tls-certs\") pod \"auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw\" (UID: \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\") " pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"
Jul 14 20:28:38.359510 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:38.359393    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-model-cache\") pod \"auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw\" (UID: \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\") " pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"
Jul 14 20:28:38.460364 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:38.460324    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-model-cache\") pod \"auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw\" (UID: \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\") " pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"
Jul 14 20:28:38.460573 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:38.460396    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-dshm\") pod \"auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw\" (UID: \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\") " pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"
Jul 14 20:28:38.460573 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:38.460420    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-tmp-dir\") pod \"auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw\" (UID: \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\") " pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"
Jul 14 20:28:38.460573 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:38.460439    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj8xw\" (UniqueName: \"kubernetes.io/projected/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-kube-api-access-hj8xw\") pod \"auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw\" (UID: \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\") " pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"
Jul 14 20:28:38.460573 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:38.460468    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-home\") pod \"auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw\" (UID: \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\") " pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"
Jul 14 20:28:38.460573 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:38.460492    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-tls-certs\") pod \"auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw\" (UID: \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\") " pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"
Jul 14 20:28:38.460876 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:38.460803    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-model-cache\") pod \"auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw\" (UID: \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\") " pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"
Jul 14 20:28:38.460876 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:38.460832    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-tmp-dir\") pod \"auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw\" (UID: \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\") " pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"
Jul 14 20:28:38.460944 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:38.460894    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-home\") pod \"auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw\" (UID: \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\") " pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"
Jul 14 20:28:38.462814 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:38.462789    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-dshm\") pod \"auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw\" (UID: \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\") " pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"
Jul 14 20:28:38.463079 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:38.463062    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-tls-certs\") pod \"auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw\" (UID: \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\") " pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"
Jul 14 20:28:38.468847 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:38.468829    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj8xw\" (UniqueName: \"kubernetes.io/projected/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-kube-api-access-hj8xw\") pod \"auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw\" (UID: \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\") " pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"
Jul 14 20:28:38.558997 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:38.558965    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"
Jul 14 20:28:38.697035 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:38.697010    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"]
Jul 14 20:28:38.699244 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:28:38.699220    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod042a6b67_7d45_45ff_8ad8_33ad6b71e1a8.slice/crio-aecdffc8f819492fff764af9821a0052162c7e3db8534e18eab946d3c7757fe6 WatchSource:0}: Error finding container aecdffc8f819492fff764af9821a0052162c7e3db8534e18eab946d3c7757fe6: Status 404 returned error can't find the container with id aecdffc8f819492fff764af9821a0052162c7e3db8534e18eab946d3c7757fe6
Jul 14 20:28:39.691023 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:39.690989    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw" event={"ID":"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8","Type":"ContainerStarted","Data":"b412cca58eb7b0fd33cde32c4f754740178ba21951a0185749ffab6151a6291e"}
Jul 14 20:28:39.691442 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:39.691037    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw" event={"ID":"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8","Type":"ContainerStarted","Data":"aecdffc8f819492fff764af9821a0052162c7e3db8534e18eab946d3c7757fe6"}
Jul 14 20:28:39.708566 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:39.708516    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw" podStartSLOduration=1.708501383 podStartE2EDuration="1.708501383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:28:39.706538062 +0000 UTC m=+2472.306841766" watchObservedRunningTime="2026-07-14 20:28:39.708501383 +0000 UTC m=+2472.308805086"
Jul 14 20:28:48.559560 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:48.559525    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"
Jul 14 20:28:48.559999 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:48.559732    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"
Jul 14 20:28:48.571859 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:48.571840    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"
Jul 14 20:28:48.749274 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:28:48.749250    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"
Jul 14 20:30:30.453551 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:30.453511    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"]
Jul 14 20:30:45.473956 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:45.473915    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw" podUID="042a6b67-7d45-45ff-8ad8-33ad6b71e1a8" containerName="main" containerID="cri-o://b412cca58eb7b0fd33cde32c4f754740178ba21951a0185749ffab6151a6291e" gracePeriod=45
Jul 14 20:30:45.728639 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:45.728581    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"
Jul 14 20:30:45.809119 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:45.809079    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-tls-certs\") pod \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\" (UID: \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\") "
Jul 14 20:30:45.809332 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:45.809188    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-tmp-dir\") pod \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\" (UID: \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\") "
Jul 14 20:30:45.809332 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:45.809214    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hj8xw\" (UniqueName: \"kubernetes.io/projected/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-kube-api-access-hj8xw\") pod \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\" (UID: \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\") "
Jul 14 20:30:45.809332 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:45.809252    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-home\") pod \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\" (UID: \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\") "
Jul 14 20:30:45.809332 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:45.809303    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-dshm\") pod \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\" (UID: \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\") "
Jul 14 20:30:45.809544 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:45.809368    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-model-cache\") pod \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\" (UID: \"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8\") "
Jul 14 20:30:45.809544 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:45.809427    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "042a6b67-7d45-45ff-8ad8-33ad6b71e1a8" (UID: "042a6b67-7d45-45ff-8ad8-33ad6b71e1a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 20:30:45.809659 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:45.809553    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-home" (OuterVolumeSpecName: "home") pod "042a6b67-7d45-45ff-8ad8-33ad6b71e1a8" (UID: "042a6b67-7d45-45ff-8ad8-33ad6b71e1a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 20:30:45.809721 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:45.809691    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-model-cache" (OuterVolumeSpecName: "model-cache") pod "042a6b67-7d45-45ff-8ad8-33ad6b71e1a8" (UID: "042a6b67-7d45-45ff-8ad8-33ad6b71e1a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 20:30:45.809986 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:45.809857    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-tmp-dir\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:30:45.809986 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:45.809883    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-home\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:30:45.809986 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:45.809898    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-model-cache\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:30:45.811366 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:45.811341    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "042a6b67-7d45-45ff-8ad8-33ad6b71e1a8" (UID: "042a6b67-7d45-45ff-8ad8-33ad6b71e1a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 20:30:45.811473 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:45.811456    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-dshm" (OuterVolumeSpecName: "dshm") pod "042a6b67-7d45-45ff-8ad8-33ad6b71e1a8" (UID: "042a6b67-7d45-45ff-8ad8-33ad6b71e1a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 20:30:45.811539 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:45.811500    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-kube-api-access-hj8xw" (OuterVolumeSpecName: "kube-api-access-hj8xw") pod "042a6b67-7d45-45ff-8ad8-33ad6b71e1a8" (UID: "042a6b67-7d45-45ff-8ad8-33ad6b71e1a8"). InnerVolumeSpecName "kube-api-access-hj8xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 20:30:45.911120 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:45.911093    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-dshm\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:30:45.911259 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:45.911127    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-tls-certs\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:30:45.911259 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:45.911139    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj8xw\" (UniqueName: \"kubernetes.io/projected/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8-kube-api-access-hj8xw\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:30:46.355476 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:46.355440    2560 generic.go:356] "Generic (PLEG): container finished" podID="042a6b67-7d45-45ff-8ad8-33ad6b71e1a8" containerID="b412cca58eb7b0fd33cde32c4f754740178ba21951a0185749ffab6151a6291e" exitCode=0
Jul 14 20:30:46.355661 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:46.355486    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"
Jul 14 20:30:46.355661 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:46.355519    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw" event={"ID":"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8","Type":"ContainerDied","Data":"b412cca58eb7b0fd33cde32c4f754740178ba21951a0185749ffab6151a6291e"}
Jul 14 20:30:46.355661 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:46.355562    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw" event={"ID":"042a6b67-7d45-45ff-8ad8-33ad6b71e1a8","Type":"ContainerDied","Data":"aecdffc8f819492fff764af9821a0052162c7e3db8534e18eab946d3c7757fe6"}
Jul 14 20:30:46.355661 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:46.355580    2560 scope.go:117] "RemoveContainer" containerID="b412cca58eb7b0fd33cde32c4f754740178ba21951a0185749ffab6151a6291e"
Jul 14 20:30:46.364728 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:46.364713    2560 scope.go:117] "RemoveContainer" containerID="b412cca58eb7b0fd33cde32c4f754740178ba21951a0185749ffab6151a6291e"
Jul 14 20:30:46.364946 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:30:46.364928    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b412cca58eb7b0fd33cde32c4f754740178ba21951a0185749ffab6151a6291e\": container with ID starting with b412cca58eb7b0fd33cde32c4f754740178ba21951a0185749ffab6151a6291e not found: ID does not exist" containerID="b412cca58eb7b0fd33cde32c4f754740178ba21951a0185749ffab6151a6291e"
Jul 14 20:30:46.364998 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:46.364955    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b412cca58eb7b0fd33cde32c4f754740178ba21951a0185749ffab6151a6291e"} err="failed to get container status \"b412cca58eb7b0fd33cde32c4f754740178ba21951a0185749ffab6151a6291e\": rpc error: code = NotFound desc = could not find container \"b412cca58eb7b0fd33cde32c4f754740178ba21951a0185749ffab6151a6291e\": container with ID starting with b412cca58eb7b0fd33cde32c4f754740178ba21951a0185749ffab6151a6291e not found: ID does not exist"
Jul 14 20:30:46.378158 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:46.378135    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"]
Jul 14 20:30:46.384728 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:46.384705    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d04b66/auth-test-model-d04b66-kserve-7bbf77cfff-tpkcw"]
Jul 14 20:30:47.889975 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:47.889938    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="042a6b67-7d45-45ff-8ad8-33ad6b71e1a8" path="/var/lib/kubelet/pods/042a6b67-7d45-45ff-8ad8-33ad6b71e1a8/volumes"
Jul 14 20:30:48.073128 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.073092    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"]
Jul 14 20:30:48.073492 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.073480    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="042a6b67-7d45-45ff-8ad8-33ad6b71e1a8" containerName="main"
Jul 14 20:30:48.073536 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.073494    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="042a6b67-7d45-45ff-8ad8-33ad6b71e1a8" containerName="main"
Jul 14 20:30:48.073596 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.073587    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="042a6b67-7d45-45ff-8ad8-33ad6b71e1a8" containerName="main"
Jul 14 20:30:48.077833 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.077817    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"
Jul 14 20:30:48.081221 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.081193    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-10ef4d\"/\"kube-root-ca.crt\""
Jul 14 20:30:48.081221 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.081207    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-10ef4d\"/\"sub-test-model-10ef4d-kserve-self-signed-certs\""
Jul 14 20:30:48.081412 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.081216    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-10ef4d\"/\"openshift-service-ca.crt\""
Jul 14 20:30:48.081412 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.081192    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-10ef4d\"/\"default-dockercfg-6jznz\""
Jul 14 20:30:48.093495 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.093467    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"]
Jul 14 20:30:48.130706 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.130670    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff0b193-21ee-40d3-8341-e7401a7f43c5-home\") pod \"sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j\" (UID: \"eff0b193-21ee-40d3-8341-e7401a7f43c5\") " pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"
Jul 14 20:30:48.130856 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.130715    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff0b193-21ee-40d3-8341-e7401a7f43c5-model-cache\") pod \"sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j\" (UID: \"eff0b193-21ee-40d3-8341-e7401a7f43c5\") " pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"
Jul 14 20:30:48.130856 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.130786    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24wvw\" (UniqueName: \"kubernetes.io/projected/eff0b193-21ee-40d3-8341-e7401a7f43c5-kube-api-access-24wvw\") pod \"sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j\" (UID: \"eff0b193-21ee-40d3-8341-e7401a7f43c5\") " pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"
Jul 14 20:30:48.130856 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.130845    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff0b193-21ee-40d3-8341-e7401a7f43c5-tmp-dir\") pod \"sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j\" (UID: \"eff0b193-21ee-40d3-8341-e7401a7f43c5\") " pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"
Jul 14 20:30:48.130979 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.130879    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff0b193-21ee-40d3-8341-e7401a7f43c5-dshm\") pod \"sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j\" (UID: \"eff0b193-21ee-40d3-8341-e7401a7f43c5\") " pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"
Jul 14 20:30:48.130979 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.130909    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff0b193-21ee-40d3-8341-e7401a7f43c5-tls-certs\") pod \"sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j\" (UID: \"eff0b193-21ee-40d3-8341-e7401a7f43c5\") " pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"
Jul 14 20:30:48.232211 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.232062    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24wvw\" (UniqueName: \"kubernetes.io/projected/eff0b193-21ee-40d3-8341-e7401a7f43c5-kube-api-access-24wvw\") pod \"sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j\" (UID: \"eff0b193-21ee-40d3-8341-e7401a7f43c5\") " pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"
Jul 14 20:30:48.232211 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.232123    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff0b193-21ee-40d3-8341-e7401a7f43c5-tmp-dir\") pod \"sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j\" (UID: \"eff0b193-21ee-40d3-8341-e7401a7f43c5\") " pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"
Jul 14 20:30:48.232211 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.232196    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff0b193-21ee-40d3-8341-e7401a7f43c5-dshm\") pod \"sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j\" (UID: \"eff0b193-21ee-40d3-8341-e7401a7f43c5\") " pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"
Jul 14 20:30:48.232512 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.232232    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff0b193-21ee-40d3-8341-e7401a7f43c5-tls-certs\") pod \"sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j\" (UID: \"eff0b193-21ee-40d3-8341-e7401a7f43c5\") " pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"
Jul 14 20:30:48.232512 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.232301    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff0b193-21ee-40d3-8341-e7401a7f43c5-home\") pod \"sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j\" (UID: \"eff0b193-21ee-40d3-8341-e7401a7f43c5\") " pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"
Jul 14 20:30:48.232512 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.232334    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff0b193-21ee-40d3-8341-e7401a7f43c5-model-cache\") pod \"sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j\" (UID: \"eff0b193-21ee-40d3-8341-e7401a7f43c5\") " pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"
Jul 14 20:30:48.232683 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.232575    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff0b193-21ee-40d3-8341-e7401a7f43c5-tmp-dir\") pod \"sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j\" (UID: \"eff0b193-21ee-40d3-8341-e7401a7f43c5\") " pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"
Jul 14 20:30:48.232683 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.232651    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff0b193-21ee-40d3-8341-e7401a7f43c5-home\") pod \"sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j\" (UID: \"eff0b193-21ee-40d3-8341-e7401a7f43c5\") " pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"
Jul 14 20:30:48.232774 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.232704    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff0b193-21ee-40d3-8341-e7401a7f43c5-model-cache\") pod \"sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j\" (UID: \"eff0b193-21ee-40d3-8341-e7401a7f43c5\") " pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"
Jul 14 20:30:48.234601 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.234575    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff0b193-21ee-40d3-8341-e7401a7f43c5-dshm\") pod \"sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j\" (UID: \"eff0b193-21ee-40d3-8341-e7401a7f43c5\") " pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"
Jul 14 20:30:48.234717 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.234700    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff0b193-21ee-40d3-8341-e7401a7f43c5-tls-certs\") pod \"sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j\" (UID: \"eff0b193-21ee-40d3-8341-e7401a7f43c5\") " pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"
Jul 14 20:30:48.240114 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.240092    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24wvw\" (UniqueName: \"kubernetes.io/projected/eff0b193-21ee-40d3-8341-e7401a7f43c5-kube-api-access-24wvw\") pod \"sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j\" (UID: \"eff0b193-21ee-40d3-8341-e7401a7f43c5\") " pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"
Jul 14 20:30:48.386711 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.386678    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"
Jul 14 20:30:48.520390 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:48.520367    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"]
Jul 14 20:30:48.522323 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:30:48.522294    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeff0b193_21ee_40d3_8341_e7401a7f43c5.slice/crio-8777edbce1b34ff2e762ba77c104891d7e633fe67307838aedf3dfb14c885fd6 WatchSource:0}: Error finding container 8777edbce1b34ff2e762ba77c104891d7e633fe67307838aedf3dfb14c885fd6: Status 404 returned error can't find the container with id 8777edbce1b34ff2e762ba77c104891d7e633fe67307838aedf3dfb14c885fd6
Jul 14 20:30:49.373494 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:49.373430    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j" event={"ID":"eff0b193-21ee-40d3-8341-e7401a7f43c5","Type":"ContainerStarted","Data":"e4d1baa9619127915d93ec2144a677c5ffdda5740d8c875f6d8066ec8cb79aa9"}
Jul 14 20:30:49.373494 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:49.373494    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j" event={"ID":"eff0b193-21ee-40d3-8341-e7401a7f43c5","Type":"ContainerStarted","Data":"8777edbce1b34ff2e762ba77c104891d7e633fe67307838aedf3dfb14c885fd6"}
Jul 14 20:30:49.393014 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:49.392954    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j" podStartSLOduration=1.392932539 podStartE2EDuration="1.392932539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:30:49.391113164 +0000 UTC m=+2601.991416868" watchObservedRunningTime="2026-07-14 20:30:49.392932539 +0000 UTC m=+2601.993236244"
Jul 14 20:30:58.387790 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:58.387748    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"
Jul 14 20:30:58.387790 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:58.387801    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"
Jul 14 20:30:58.400559 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:58.400526    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"
Jul 14 20:30:58.432376 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:30:58.432350    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"
Jul 14 20:31:49.730975 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:49.730940    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"]
Jul 14 20:31:58.050100 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:58.050067    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"]
Jul 14 20:31:58.053601 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:58.053584    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"
Jul 14 20:31:58.055972 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:58.055945    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-10ef4d\"/\"rate-test-model-10ef4d-kserve-self-signed-certs\""
Jul 14 20:31:58.061727 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:58.061707    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"]
Jul 14 20:31:58.169877 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:58.169841    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkjf8\" (UniqueName: \"kubernetes.io/projected/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-kube-api-access-bkjf8\") pod \"rate-test-model-10ef4d-kserve-684456675f-p9lrc\" (UID: \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\") " pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"
Jul 14 20:31:58.169877 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:58.169890    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-model-cache\") pod \"rate-test-model-10ef4d-kserve-684456675f-p9lrc\" (UID: \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\") " pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"
Jul 14 20:31:58.170112 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:58.169971    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-tmp-dir\") pod \"rate-test-model-10ef4d-kserve-684456675f-p9lrc\" (UID: \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\") " pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"
Jul 14 20:31:58.170112 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:58.170038    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-tls-certs\") pod \"rate-test-model-10ef4d-kserve-684456675f-p9lrc\" (UID: \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\") " pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"
Jul 14 20:31:58.170112 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:58.170089    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-home\") pod \"rate-test-model-10ef4d-kserve-684456675f-p9lrc\" (UID: \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\") " pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"
Jul 14 20:31:58.170272 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:58.170113    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-dshm\") pod \"rate-test-model-10ef4d-kserve-684456675f-p9lrc\" (UID: \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\") " pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"
Jul 14 20:31:58.271360 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:58.271317    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-tmp-dir\") pod \"rate-test-model-10ef4d-kserve-684456675f-p9lrc\" (UID: \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\") " pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"
Jul 14 20:31:58.271590 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:58.271370    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-tls-certs\") pod \"rate-test-model-10ef4d-kserve-684456675f-p9lrc\" (UID: \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\") " pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"
Jul 14 20:31:58.271590 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:58.271407    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-home\") pod \"rate-test-model-10ef4d-kserve-684456675f-p9lrc\" (UID: \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\") " pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"
Jul 14 20:31:58.271590 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:58.271434    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-dshm\") pod \"rate-test-model-10ef4d-kserve-684456675f-p9lrc\" (UID: \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\") " pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"
Jul 14 20:31:58.271590 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:58.271512    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkjf8\" (UniqueName: \"kubernetes.io/projected/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-kube-api-access-bkjf8\") pod \"rate-test-model-10ef4d-kserve-684456675f-p9lrc\" (UID: \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\") " pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"
Jul 14 20:31:58.271590 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:58.271542    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-model-cache\") pod \"rate-test-model-10ef4d-kserve-684456675f-p9lrc\" (UID: \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\") " pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"
Jul 14 20:31:58.271890 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:58.271774    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-tmp-dir\") pod \"rate-test-model-10ef4d-kserve-684456675f-p9lrc\" (UID: \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\") " pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"
Jul 14 20:31:58.271890 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:58.271812    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-home\") pod \"rate-test-model-10ef4d-kserve-684456675f-p9lrc\" (UID: \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\") " pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"
Jul 14 20:31:58.272013 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:58.271916    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-model-cache\") pod \"rate-test-model-10ef4d-kserve-684456675f-p9lrc\" (UID: \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\") " pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"
Jul 14 20:31:58.273783 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:58.273764    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-dshm\") pod \"rate-test-model-10ef4d-kserve-684456675f-p9lrc\" (UID: \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\") " pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"
Jul 14 20:31:58.274034 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:58.274015    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-tls-certs\") pod \"rate-test-model-10ef4d-kserve-684456675f-p9lrc\" (UID: \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\") " pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"
Jul 14 20:31:58.279874 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:58.279851    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkjf8\" (UniqueName: \"kubernetes.io/projected/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-kube-api-access-bkjf8\") pod \"rate-test-model-10ef4d-kserve-684456675f-p9lrc\" (UID: \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\") " pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"
Jul 14 20:31:58.363369 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:58.363285    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"
Jul 14 20:31:58.497506 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:58.497482    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"]
Jul 14 20:31:58.499295 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:31:58.499268    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9618d63d_5e4e_4ed1_aea8_2f5c0b0283f5.slice/crio-326bc892e4747fa80f0ec6661af69dbccf04136f9b7fba5e174f0c026251c3e0 WatchSource:0}: Error finding container 326bc892e4747fa80f0ec6661af69dbccf04136f9b7fba5e174f0c026251c3e0: Status 404 returned error can't find the container with id 326bc892e4747fa80f0ec6661af69dbccf04136f9b7fba5e174f0c026251c3e0
Jul 14 20:31:58.734685 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:58.734596    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc" event={"ID":"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5","Type":"ContainerStarted","Data":"98f00eadda64b29e8f7ea704037118e9f24173f396df4240fc47878273911fc1"}
Jul 14 20:31:58.734685 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:58.734636    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc" event={"ID":"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5","Type":"ContainerStarted","Data":"326bc892e4747fa80f0ec6661af69dbccf04136f9b7fba5e174f0c026251c3e0"}
Jul 14 20:31:58.752288 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:31:58.752227    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc" podStartSLOduration=0.75220842 podStartE2EDuration="752.20842ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:31:58.751586729 +0000 UTC m=+2671.351890433" watchObservedRunningTime="2026-07-14 20:31:58.75220842 +0000 UTC m=+2671.352512124"
Jul 14 20:32:04.751743 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:04.751674    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j" podUID="eff0b193-21ee-40d3-8341-e7401a7f43c5" containerName="main" containerID="cri-o://e4d1baa9619127915d93ec2144a677c5ffdda5740d8c875f6d8066ec8cb79aa9" gracePeriod=45
Jul 14 20:32:04.999795 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:04.999768    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"
Jul 14 20:32:05.135288 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.135247    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff0b193-21ee-40d3-8341-e7401a7f43c5-home\") pod \"eff0b193-21ee-40d3-8341-e7401a7f43c5\" (UID: \"eff0b193-21ee-40d3-8341-e7401a7f43c5\") "
Jul 14 20:32:05.135492 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.135319    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff0b193-21ee-40d3-8341-e7401a7f43c5-model-cache\") pod \"eff0b193-21ee-40d3-8341-e7401a7f43c5\" (UID: \"eff0b193-21ee-40d3-8341-e7401a7f43c5\") "
Jul 14 20:32:05.135492 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.135341    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff0b193-21ee-40d3-8341-e7401a7f43c5-tmp-dir\") pod \"eff0b193-21ee-40d3-8341-e7401a7f43c5\" (UID: \"eff0b193-21ee-40d3-8341-e7401a7f43c5\") "
Jul 14 20:32:05.135492 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.135363    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff0b193-21ee-40d3-8341-e7401a7f43c5-dshm\") pod \"eff0b193-21ee-40d3-8341-e7401a7f43c5\" (UID: \"eff0b193-21ee-40d3-8341-e7401a7f43c5\") "
Jul 14 20:32:05.135492 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.135383    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24wvw\" (UniqueName: \"kubernetes.io/projected/eff0b193-21ee-40d3-8341-e7401a7f43c5-kube-api-access-24wvw\") pod \"eff0b193-21ee-40d3-8341-e7401a7f43c5\" (UID: \"eff0b193-21ee-40d3-8341-e7401a7f43c5\") "
Jul 14 20:32:05.135492 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.135426    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff0b193-21ee-40d3-8341-e7401a7f43c5-tls-certs\") pod \"eff0b193-21ee-40d3-8341-e7401a7f43c5\" (UID: \"eff0b193-21ee-40d3-8341-e7401a7f43c5\") "
Jul 14 20:32:05.135742 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.135586    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff0b193-21ee-40d3-8341-e7401a7f43c5-home" (OuterVolumeSpecName: "home") pod "eff0b193-21ee-40d3-8341-e7401a7f43c5" (UID: "eff0b193-21ee-40d3-8341-e7401a7f43c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 20:32:05.135742 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.135641    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff0b193-21ee-40d3-8341-e7401a7f43c5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eff0b193-21ee-40d3-8341-e7401a7f43c5" (UID: "eff0b193-21ee-40d3-8341-e7401a7f43c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 20:32:05.135742 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.135692    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff0b193-21ee-40d3-8341-e7401a7f43c5-model-cache" (OuterVolumeSpecName: "model-cache") pod "eff0b193-21ee-40d3-8341-e7401a7f43c5" (UID: "eff0b193-21ee-40d3-8341-e7401a7f43c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 20:32:05.135882 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.135774    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff0b193-21ee-40d3-8341-e7401a7f43c5-model-cache\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:32:05.135882 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.135812    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff0b193-21ee-40d3-8341-e7401a7f43c5-tmp-dir\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:32:05.135882 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.135826    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff0b193-21ee-40d3-8341-e7401a7f43c5-home\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:32:05.137913 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.137873    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eff0b193-21ee-40d3-8341-e7401a7f43c5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eff0b193-21ee-40d3-8341-e7401a7f43c5" (UID: "eff0b193-21ee-40d3-8341-e7401a7f43c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 20:32:05.137913 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.137883    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff0b193-21ee-40d3-8341-e7401a7f43c5-dshm" (OuterVolumeSpecName: "dshm") pod "eff0b193-21ee-40d3-8341-e7401a7f43c5" (UID: "eff0b193-21ee-40d3-8341-e7401a7f43c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 20:32:05.138063 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.137927    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff0b193-21ee-40d3-8341-e7401a7f43c5-kube-api-access-24wvw" (OuterVolumeSpecName: "kube-api-access-24wvw") pod "eff0b193-21ee-40d3-8341-e7401a7f43c5" (UID: "eff0b193-21ee-40d3-8341-e7401a7f43c5"). InnerVolumeSpecName "kube-api-access-24wvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 20:32:05.237244 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.237209    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff0b193-21ee-40d3-8341-e7401a7f43c5-dshm\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:32:05.237244 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.237238    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24wvw\" (UniqueName: \"kubernetes.io/projected/eff0b193-21ee-40d3-8341-e7401a7f43c5-kube-api-access-24wvw\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:32:05.237244 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.237248    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff0b193-21ee-40d3-8341-e7401a7f43c5-tls-certs\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:32:05.779489 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.779456    2560 generic.go:356] "Generic (PLEG): container finished" podID="eff0b193-21ee-40d3-8341-e7401a7f43c5" containerID="e4d1baa9619127915d93ec2144a677c5ffdda5740d8c875f6d8066ec8cb79aa9" exitCode=0
Jul 14 20:32:05.779911 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.779503    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"
Jul 14 20:32:05.779911 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.779545    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j" event={"ID":"eff0b193-21ee-40d3-8341-e7401a7f43c5","Type":"ContainerDied","Data":"e4d1baa9619127915d93ec2144a677c5ffdda5740d8c875f6d8066ec8cb79aa9"}
Jul 14 20:32:05.779911 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.779585    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j" event={"ID":"eff0b193-21ee-40d3-8341-e7401a7f43c5","Type":"ContainerDied","Data":"8777edbce1b34ff2e762ba77c104891d7e633fe67307838aedf3dfb14c885fd6"}
Jul 14 20:32:05.779911 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.779601    2560 scope.go:117] "RemoveContainer" containerID="e4d1baa9619127915d93ec2144a677c5ffdda5740d8c875f6d8066ec8cb79aa9"
Jul 14 20:32:05.789664 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.789646    2560 scope.go:117] "RemoveContainer" containerID="e4d1baa9619127915d93ec2144a677c5ffdda5740d8c875f6d8066ec8cb79aa9"
Jul 14 20:32:05.789942 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:32:05.789914    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4d1baa9619127915d93ec2144a677c5ffdda5740d8c875f6d8066ec8cb79aa9\": container with ID starting with e4d1baa9619127915d93ec2144a677c5ffdda5740d8c875f6d8066ec8cb79aa9 not found: ID does not exist" containerID="e4d1baa9619127915d93ec2144a677c5ffdda5740d8c875f6d8066ec8cb79aa9"
Jul 14 20:32:05.790007 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.789950    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4d1baa9619127915d93ec2144a677c5ffdda5740d8c875f6d8066ec8cb79aa9"} err="failed to get container status \"e4d1baa9619127915d93ec2144a677c5ffdda5740d8c875f6d8066ec8cb79aa9\": rpc error: code = NotFound desc = could not find container \"e4d1baa9619127915d93ec2144a677c5ffdda5740d8c875f6d8066ec8cb79aa9\": container with ID starting with e4d1baa9619127915d93ec2144a677c5ffdda5740d8c875f6d8066ec8cb79aa9 not found: ID does not exist"
Jul 14 20:32:05.801780 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.801756    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"]
Jul 14 20:32:05.805589 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.805571    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-10ef4d/sub-test-model-10ef4d-kserve-6cf486f7f5-7t54j"]
Jul 14 20:32:05.890433 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:05.890400    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eff0b193-21ee-40d3-8341-e7401a7f43c5" path="/var/lib/kubelet/pods/eff0b193-21ee-40d3-8341-e7401a7f43c5/volumes"
Jul 14 20:32:08.363570 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:08.363535    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"
Jul 14 20:32:08.363972 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:08.363611    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"
Jul 14 20:32:08.375824 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:08.375801    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"
Jul 14 20:32:08.816775 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:08.816747    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"
Jul 14 20:32:28.051464 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:28.051390    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tzr6_4a36af4c-9eac-465f-9a0e-d65a382530f7/ovn-acl-logging/0.log"
Jul 14 20:32:28.062607 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:32:28.062588    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8tzr6_4a36af4c-9eac-465f-9a0e-d65a382530f7/ovn-acl-logging/0.log"
Jul 14 20:33:08.429493 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:08.429460    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"]
Jul 14 20:33:10.404758 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:10.404680    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7704-140546066910384"]
Jul 14 20:33:10.405120 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:10.405053    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eff0b193-21ee-40d3-8341-e7401a7f43c5" containerName="main"
Jul 14 20:33:10.405120 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:10.405065    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eff0b193-21ee-40d3-8341-e7401a7f43c5" containerName="main"
Jul 14 20:33:10.405211 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:10.405141    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eff0b193-21ee-40d3-8341-e7401a7f43c5" containerName="main"
Jul 14 20:33:10.415490 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:10.415459    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7704-140546066910384"]
Jul 14 20:33:10.415666 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:10.415545    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-140546066910384"
Jul 14 20:33:10.417807 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:10.417782    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p2ln2\""
Jul 14 20:33:10.513797 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:10.513763    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfjwv\" (UniqueName: \"kubernetes.io/projected/e361af8e-4366-42cc-b266-d4820f274dd2-kube-api-access-xfjwv\") pod \"test-curl-7704-140546066910384\" (UID: \"e361af8e-4366-42cc-b266-d4820f274dd2\") " pod="opendatahub/test-curl-7704-140546066910384"
Jul 14 20:33:10.615088 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:10.615055    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfjwv\" (UniqueName: \"kubernetes.io/projected/e361af8e-4366-42cc-b266-d4820f274dd2-kube-api-access-xfjwv\") pod \"test-curl-7704-140546066910384\" (UID: \"e361af8e-4366-42cc-b266-d4820f274dd2\") " pod="opendatahub/test-curl-7704-140546066910384"
Jul 14 20:33:10.624374 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:10.624338    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfjwv\" (UniqueName: \"kubernetes.io/projected/e361af8e-4366-42cc-b266-d4820f274dd2-kube-api-access-xfjwv\") pod \"test-curl-7704-140546066910384\" (UID: \"e361af8e-4366-42cc-b266-d4820f274dd2\") " pod="opendatahub/test-curl-7704-140546066910384"
Jul 14 20:33:10.725247 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:10.725145    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-140546066910384"
Jul 14 20:33:10.855341 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:10.855318    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7704-140546066910384"]
Jul 14 20:33:10.857288 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:33:10.857261    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode361af8e_4366_42cc_b266_d4820f274dd2.slice/crio-4e516ce54be8326d7bc3912b26ec6f6bef59e9fe317dfed70c1ff02beb550fea WatchSource:0}: Error finding container 4e516ce54be8326d7bc3912b26ec6f6bef59e9fe317dfed70c1ff02beb550fea: Status 404 returned error can't find the container with id 4e516ce54be8326d7bc3912b26ec6f6bef59e9fe317dfed70c1ff02beb550fea
Jul 14 20:33:10.858702 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:10.858684    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 20:33:11.120204 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:11.120154    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7704-140546066910384" event={"ID":"e361af8e-4366-42cc-b266-d4820f274dd2","Type":"ContainerStarted","Data":"4e516ce54be8326d7bc3912b26ec6f6bef59e9fe317dfed70c1ff02beb550fea"}
Jul 14 20:33:12.126946 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:12.126908    2560 generic.go:356] "Generic (PLEG): container finished" podID="e361af8e-4366-42cc-b266-d4820f274dd2" containerID="4123e3ca057e5066f6be7c4cebf1d764403e3219fe35ac1a99a82d2ec733588d" exitCode=0
Jul 14 20:33:12.127361 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:12.126997    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7704-140546066910384" event={"ID":"e361af8e-4366-42cc-b266-d4820f274dd2","Type":"ContainerDied","Data":"4123e3ca057e5066f6be7c4cebf1d764403e3219fe35ac1a99a82d2ec733588d"}
Jul 14 20:33:13.266032 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:13.266004    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-140546066910384"
Jul 14 20:33:13.298596 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:13.298563    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7704-140546066910384_e361af8e-4366-42cc-b266-d4820f274dd2/test-curl-7704-140546066910384/0.log"
Jul 14 20:33:13.323415 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:13.323384    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7704-140546066910384"]
Jul 14 20:33:13.325315 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:13.325297    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7704-140546066910384"]
Jul 14 20:33:13.340954 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:13.340932    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfjwv\" (UniqueName: \"kubernetes.io/projected/e361af8e-4366-42cc-b266-d4820f274dd2-kube-api-access-xfjwv\") pod \"e361af8e-4366-42cc-b266-d4820f274dd2\" (UID: \"e361af8e-4366-42cc-b266-d4820f274dd2\") "
Jul 14 20:33:13.343208 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:13.343182    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e361af8e-4366-42cc-b266-d4820f274dd2-kube-api-access-xfjwv" (OuterVolumeSpecName: "kube-api-access-xfjwv") pod "e361af8e-4366-42cc-b266-d4820f274dd2" (UID: "e361af8e-4366-42cc-b266-d4820f274dd2"). InnerVolumeSpecName "kube-api-access-xfjwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 20:33:13.442334 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:13.442247    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfjwv\" (UniqueName: \"kubernetes.io/projected/e361af8e-4366-42cc-b266-d4820f274dd2-kube-api-access-xfjwv\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:33:13.889376 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:13.889342    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e361af8e-4366-42cc-b266-d4820f274dd2" path="/var/lib/kubelet/pods/e361af8e-4366-42cc-b266-d4820f274dd2/volumes"
Jul 14 20:33:14.138968 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:14.138939    2560 scope.go:117] "RemoveContainer" containerID="4123e3ca057e5066f6be7c4cebf1d764403e3219fe35ac1a99a82d2ec733588d"
Jul 14 20:33:14.139158 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:14.138944    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-140546066910384"
Jul 14 20:33:17.067748 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:17.067715    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7704-140546066850560"]
Jul 14 20:33:17.068143 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:17.068125    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e361af8e-4366-42cc-b266-d4820f274dd2" containerName="test-curl-7704-140546066910384"
Jul 14 20:33:17.068143 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:17.068138    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e361af8e-4366-42cc-b266-d4820f274dd2" containerName="test-curl-7704-140546066910384"
Jul 14 20:33:17.068252 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:17.068238    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e361af8e-4366-42cc-b266-d4820f274dd2" containerName="test-curl-7704-140546066910384"
Jul 14 20:33:17.071386 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:17.071367    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-140546066850560"
Jul 14 20:33:17.073483 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:17.073465    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p2ln2\""
Jul 14 20:33:17.078119 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:17.078096    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7704-140546066850560"]
Jul 14 20:33:17.174082 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:17.174048    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knb2g\" (UniqueName: \"kubernetes.io/projected/520a3f58-6036-4022-8771-415da96e662e-kube-api-access-knb2g\") pod \"test-curl-7704-140546066850560\" (UID: \"520a3f58-6036-4022-8771-415da96e662e\") " pod="opendatahub/test-curl-7704-140546066850560"
Jul 14 20:33:17.275366 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:17.275330    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knb2g\" (UniqueName: \"kubernetes.io/projected/520a3f58-6036-4022-8771-415da96e662e-kube-api-access-knb2g\") pod \"test-curl-7704-140546066850560\" (UID: \"520a3f58-6036-4022-8771-415da96e662e\") " pod="opendatahub/test-curl-7704-140546066850560"
Jul 14 20:33:17.284000 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:17.283961    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knb2g\" (UniqueName: \"kubernetes.io/projected/520a3f58-6036-4022-8771-415da96e662e-kube-api-access-knb2g\") pod \"test-curl-7704-140546066850560\" (UID: \"520a3f58-6036-4022-8771-415da96e662e\") " pod="opendatahub/test-curl-7704-140546066850560"
Jul 14 20:33:17.381923 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:17.381833    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-140546066850560"
Jul 14 20:33:17.523404 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:17.523267    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7704-140546066850560"]
Jul 14 20:33:17.525599 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:33:17.525574    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod520a3f58_6036_4022_8771_415da96e662e.slice/crio-cd7605ed103106da477f684f83982af6de4cd5e89b62682aa8fb614e8767bc59 WatchSource:0}: Error finding container cd7605ed103106da477f684f83982af6de4cd5e89b62682aa8fb614e8767bc59: Status 404 returned error can't find the container with id cd7605ed103106da477f684f83982af6de4cd5e89b62682aa8fb614e8767bc59
Jul 14 20:33:18.161273 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:18.161150    2560 generic.go:356] "Generic (PLEG): container finished" podID="520a3f58-6036-4022-8771-415da96e662e" containerID="96beacfca58da2b7003b797b6c80984cc156ac4803a2f243f9710c24d6085c0a" exitCode=0
Jul 14 20:33:18.161273 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:18.161248    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7704-140546066850560" event={"ID":"520a3f58-6036-4022-8771-415da96e662e","Type":"ContainerDied","Data":"96beacfca58da2b7003b797b6c80984cc156ac4803a2f243f9710c24d6085c0a"}
Jul 14 20:33:18.161680 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:18.161295    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7704-140546066850560" event={"ID":"520a3f58-6036-4022-8771-415da96e662e","Type":"ContainerStarted","Data":"cd7605ed103106da477f684f83982af6de4cd5e89b62682aa8fb614e8767bc59"}
Jul 14 20:33:19.299112 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:19.299086    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-140546066850560"
Jul 14 20:33:19.319510 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:19.319481    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7704-140546066850560_520a3f58-6036-4022-8771-415da96e662e/test-curl-7704-140546066850560/0.log"
Jul 14 20:33:19.343757 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:19.343724    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7704-140546066850560"]
Jul 14 20:33:19.347804 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:19.347777    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7704-140546066850560"]
Jul 14 20:33:19.395144 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:19.395110    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knb2g\" (UniqueName: \"kubernetes.io/projected/520a3f58-6036-4022-8771-415da96e662e-kube-api-access-knb2g\") pod \"520a3f58-6036-4022-8771-415da96e662e\" (UID: \"520a3f58-6036-4022-8771-415da96e662e\") "
Jul 14 20:33:19.397158 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:19.397120    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520a3f58-6036-4022-8771-415da96e662e-kube-api-access-knb2g" (OuterVolumeSpecName: "kube-api-access-knb2g") pod "520a3f58-6036-4022-8771-415da96e662e" (UID: "520a3f58-6036-4022-8771-415da96e662e"). InnerVolumeSpecName "kube-api-access-knb2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 20:33:19.496119 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:19.496044    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knb2g\" (UniqueName: \"kubernetes.io/projected/520a3f58-6036-4022-8771-415da96e662e-kube-api-access-knb2g\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:33:19.889268 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:19.889238    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="520a3f58-6036-4022-8771-415da96e662e" path="/var/lib/kubelet/pods/520a3f58-6036-4022-8771-415da96e662e/volumes"
Jul 14 20:33:20.172586 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:20.172505    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-140546066850560"
Jul 14 20:33:20.172586 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:20.172560    2560 scope.go:117] "RemoveContainer" containerID="96beacfca58da2b7003b797b6c80984cc156ac4803a2f243f9710c24d6085c0a"
Jul 14 20:33:23.450407 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:23.450357    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc" podUID="9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5" containerName="main" containerID="cri-o://98f00eadda64b29e8f7ea704037118e9f24173f396df4240fc47878273911fc1" gracePeriod=45
Jul 14 20:33:23.707791 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:23.707735    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"
Jul 14 20:33:23.734495 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:23.734464    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkjf8\" (UniqueName: \"kubernetes.io/projected/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-kube-api-access-bkjf8\") pod \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\" (UID: \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\") "
Jul 14 20:33:23.734495 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:23.734497    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-model-cache\") pod \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\" (UID: \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\") "
Jul 14 20:33:23.734717 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:23.734570    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-tmp-dir\") pod \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\" (UID: \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\") "
Jul 14 20:33:23.734717 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:23.734621    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-home\") pod \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\" (UID: \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\") "
Jul 14 20:33:23.734717 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:23.734668    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-dshm\") pod \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\" (UID: \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\") "
Jul 14 20:33:23.734717 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:23.734705    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-tls-certs\") pod \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\" (UID: \"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5\") "
Jul 14 20:33:23.734982 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:23.734779    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5" (UID: "9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 20:33:23.734982 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:23.734837    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5" (UID: "9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 20:33:23.734982 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:23.734919    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-home" (OuterVolumeSpecName: "home") pod "9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5" (UID: "9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 20:33:23.735151 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:23.735050    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-tmp-dir\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:33:23.735151 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:23.735071    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-home\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:33:23.735151 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:23.735085    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-model-cache\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:33:23.736876 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:23.736850    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-dshm" (OuterVolumeSpecName: "dshm") pod "9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5" (UID: "9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 20:33:23.737002 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:23.736869    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-kube-api-access-bkjf8" (OuterVolumeSpecName: "kube-api-access-bkjf8") pod "9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5" (UID: "9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5"). InnerVolumeSpecName "kube-api-access-bkjf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 20:33:23.737002 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:23.736951    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5" (UID: "9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 20:33:23.835813 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:23.835781    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkjf8\" (UniqueName: \"kubernetes.io/projected/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-kube-api-access-bkjf8\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:33:23.835813 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:23.835809    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-dshm\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:33:23.835813 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:23.835820    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5-tls-certs\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:33:24.194703 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:24.194667    2560 generic.go:356] "Generic (PLEG): container finished" podID="9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5" containerID="98f00eadda64b29e8f7ea704037118e9f24173f396df4240fc47878273911fc1" exitCode=0
Jul 14 20:33:24.194875 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:24.194714    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"
Jul 14 20:33:24.194875 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:24.194753    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc" event={"ID":"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5","Type":"ContainerDied","Data":"98f00eadda64b29e8f7ea704037118e9f24173f396df4240fc47878273911fc1"}
Jul 14 20:33:24.194875 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:24.194792    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc" event={"ID":"9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5","Type":"ContainerDied","Data":"326bc892e4747fa80f0ec6661af69dbccf04136f9b7fba5e174f0c026251c3e0"}
Jul 14 20:33:24.194875 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:24.194808    2560 scope.go:117] "RemoveContainer" containerID="98f00eadda64b29e8f7ea704037118e9f24173f396df4240fc47878273911fc1"
Jul 14 20:33:24.207278 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:24.207254    2560 scope.go:117] "RemoveContainer" containerID="98f00eadda64b29e8f7ea704037118e9f24173f396df4240fc47878273911fc1"
Jul 14 20:33:24.207562 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:33:24.207542    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98f00eadda64b29e8f7ea704037118e9f24173f396df4240fc47878273911fc1\": container with ID starting with 98f00eadda64b29e8f7ea704037118e9f24173f396df4240fc47878273911fc1 not found: ID does not exist" containerID="98f00eadda64b29e8f7ea704037118e9f24173f396df4240fc47878273911fc1"
Jul 14 20:33:24.207621 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:24.207576    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98f00eadda64b29e8f7ea704037118e9f24173f396df4240fc47878273911fc1"} err="failed to get container status \"98f00eadda64b29e8f7ea704037118e9f24173f396df4240fc47878273911fc1\": rpc error: code = NotFound desc = could not find container \"98f00eadda64b29e8f7ea704037118e9f24173f396df4240fc47878273911fc1\": container with ID starting with 98f00eadda64b29e8f7ea704037118e9f24173f396df4240fc47878273911fc1 not found: ID does not exist"
Jul 14 20:33:24.213220 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:24.213196    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"]
Jul 14 20:33:24.217591 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:24.217571    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-10ef4d/rate-test-model-10ef4d-kserve-684456675f-p9lrc"]
Jul 14 20:33:25.099334 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:25.099301    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7704-140546066851568"]
Jul 14 20:33:25.099709 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:25.099674    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5" containerName="main"
Jul 14 20:33:25.099709 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:25.099685    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5" containerName="main"
Jul 14 20:33:25.099709 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:25.099704    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="520a3f58-6036-4022-8771-415da96e662e" containerName="test-curl-7704-140546066850560"
Jul 14 20:33:25.099815 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:25.099712    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="520a3f58-6036-4022-8771-415da96e662e" containerName="test-curl-7704-140546066850560"
Jul 14 20:33:25.099815 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:25.099783    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5" containerName="main"
Jul 14 20:33:25.099815 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:25.099797    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="520a3f58-6036-4022-8771-415da96e662e" containerName="test-curl-7704-140546066850560"
Jul 14 20:33:25.104433 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:25.104418    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-140546066851568"
Jul 14 20:33:25.106834 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:25.106806    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p2ln2\""
Jul 14 20:33:25.108354 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:25.108333    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7704-140546066851568"]
Jul 14 20:33:25.146481 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:25.146443    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2knpl\" (UniqueName: \"kubernetes.io/projected/3fbbbedf-4fb9-4f2f-8ba7-508b12a620ea-kube-api-access-2knpl\") pod \"test-curl-7704-140546066851568\" (UID: \"3fbbbedf-4fb9-4f2f-8ba7-508b12a620ea\") " pod="opendatahub/test-curl-7704-140546066851568"
Jul 14 20:33:25.247004 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:25.246968    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2knpl\" (UniqueName: \"kubernetes.io/projected/3fbbbedf-4fb9-4f2f-8ba7-508b12a620ea-kube-api-access-2knpl\") pod \"test-curl-7704-140546066851568\" (UID: \"3fbbbedf-4fb9-4f2f-8ba7-508b12a620ea\") " pod="opendatahub/test-curl-7704-140546066851568"
Jul 14 20:33:25.255099 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:25.255073    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2knpl\" (UniqueName: \"kubernetes.io/projected/3fbbbedf-4fb9-4f2f-8ba7-508b12a620ea-kube-api-access-2knpl\") pod \"test-curl-7704-140546066851568\" (UID: \"3fbbbedf-4fb9-4f2f-8ba7-508b12a620ea\") " pod="opendatahub/test-curl-7704-140546066851568"
Jul 14 20:33:25.414415 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:25.414322    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-140546066851568"
Jul 14 20:33:25.545467 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:25.545444    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7704-140546066851568"]
Jul 14 20:33:25.547988 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:33:25.547952    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fbbbedf_4fb9_4f2f_8ba7_508b12a620ea.slice/crio-c63a294d23ac460043febc69f15f0f8dfff85c4dff839a3a411c49c4174f1be3 WatchSource:0}: Error finding container c63a294d23ac460043febc69f15f0f8dfff85c4dff839a3a411c49c4174f1be3: Status 404 returned error can't find the container with id c63a294d23ac460043febc69f15f0f8dfff85c4dff839a3a411c49c4174f1be3
Jul 14 20:33:25.890000 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:25.889967    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5" path="/var/lib/kubelet/pods/9618d63d-5e4e-4ed1-aea8-2f5c0b0283f5/volumes"
Jul 14 20:33:26.208569 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:26.208478    2560 generic.go:356] "Generic (PLEG): container finished" podID="3fbbbedf-4fb9-4f2f-8ba7-508b12a620ea" containerID="115612da3176bcd6f8725e1cf71a5990e5474e8bb8e0e20ceec248a529861640" exitCode=0
Jul 14 20:33:26.208569 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:26.208535    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7704-140546066851568" event={"ID":"3fbbbedf-4fb9-4f2f-8ba7-508b12a620ea","Type":"ContainerDied","Data":"115612da3176bcd6f8725e1cf71a5990e5474e8bb8e0e20ceec248a529861640"}
Jul 14 20:33:26.208569 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:26.208566    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7704-140546066851568" event={"ID":"3fbbbedf-4fb9-4f2f-8ba7-508b12a620ea","Type":"ContainerStarted","Data":"c63a294d23ac460043febc69f15f0f8dfff85c4dff839a3a411c49c4174f1be3"}
Jul 14 20:33:27.346972 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:27.346939    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-140546066851568"
Jul 14 20:33:27.367781 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:27.367752    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7704-140546066851568_3fbbbedf-4fb9-4f2f-8ba7-508b12a620ea/test-curl-7704-140546066851568/0.log"
Jul 14 20:33:27.391924 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:27.391891    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7704-140546066851568"]
Jul 14 20:33:27.395651 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:27.395630    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7704-140546066851568"]
Jul 14 20:33:27.466254 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:27.466137    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2knpl\" (UniqueName: \"kubernetes.io/projected/3fbbbedf-4fb9-4f2f-8ba7-508b12a620ea-kube-api-access-2knpl\") pod \"3fbbbedf-4fb9-4f2f-8ba7-508b12a620ea\" (UID: \"3fbbbedf-4fb9-4f2f-8ba7-508b12a620ea\") "
Jul 14 20:33:27.468357 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:27.468333    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fbbbedf-4fb9-4f2f-8ba7-508b12a620ea-kube-api-access-2knpl" (OuterVolumeSpecName: "kube-api-access-2knpl") pod "3fbbbedf-4fb9-4f2f-8ba7-508b12a620ea" (UID: "3fbbbedf-4fb9-4f2f-8ba7-508b12a620ea"). InnerVolumeSpecName "kube-api-access-2knpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 20:33:27.567618 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:27.567583    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2knpl\" (UniqueName: \"kubernetes.io/projected/3fbbbedf-4fb9-4f2f-8ba7-508b12a620ea-kube-api-access-2knpl\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:33:27.892243 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:27.892215    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fbbbedf-4fb9-4f2f-8ba7-508b12a620ea" path="/var/lib/kubelet/pods/3fbbbedf-4fb9-4f2f-8ba7-508b12a620ea/volumes"
Jul 14 20:33:28.221508 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:28.221426    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-140546066851568"
Jul 14 20:33:28.221508 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:28.221471    2560 scope.go:117] "RemoveContainer" containerID="115612da3176bcd6f8725e1cf71a5990e5474e8bb8e0e20ceec248a529861640"
Jul 14 20:33:38.368003 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:38.367961    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj"]
Jul 14 20:33:38.368500 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:38.368282    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj" podUID="7705d7ef-f85a-4ada-b3d1-842e234be770" containerName="maas-api" containerID="cri-o://591b1bc8647c4e2be7f6b5cad163aa856054f7985734ddbac7cdba9f99a8e415" gracePeriod=30
Jul 14 20:33:38.506821 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:38.506771    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj" podUID="7705d7ef-f85a-4ada-b3d1-842e234be770" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.61:8443/health\": dial tcp 10.133.0.61:8443: connect: connection refused"
Jul 14 20:33:38.815455 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:38.815427    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj"
Jul 14 20:33:38.872493 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:38.872458    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knwqr\" (UniqueName: \"kubernetes.io/projected/7705d7ef-f85a-4ada-b3d1-842e234be770-kube-api-access-knwqr\") pod \"7705d7ef-f85a-4ada-b3d1-842e234be770\" (UID: \"7705d7ef-f85a-4ada-b3d1-842e234be770\") "
Jul 14 20:33:38.872653 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:38.872542    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7705d7ef-f85a-4ada-b3d1-842e234be770-maas-api-tls\") pod \"7705d7ef-f85a-4ada-b3d1-842e234be770\" (UID: \"7705d7ef-f85a-4ada-b3d1-842e234be770\") "
Jul 14 20:33:38.872653 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:38.872634    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7705d7ef-f85a-4ada-b3d1-842e234be770-openshift-service-ca\") pod \"7705d7ef-f85a-4ada-b3d1-842e234be770\" (UID: \"7705d7ef-f85a-4ada-b3d1-842e234be770\") "
Jul 14 20:33:38.872991 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:38.872964    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7705d7ef-f85a-4ada-b3d1-842e234be770-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7705d7ef-f85a-4ada-b3d1-842e234be770" (UID: "7705d7ef-f85a-4ada-b3d1-842e234be770"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 20:33:38.874640 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:38.874617    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7705d7ef-f85a-4ada-b3d1-842e234be770-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7705d7ef-f85a-4ada-b3d1-842e234be770" (UID: "7705d7ef-f85a-4ada-b3d1-842e234be770"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 20:33:38.874700 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:38.874660    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7705d7ef-f85a-4ada-b3d1-842e234be770-kube-api-access-knwqr" (OuterVolumeSpecName: "kube-api-access-knwqr") pod "7705d7ef-f85a-4ada-b3d1-842e234be770" (UID: "7705d7ef-f85a-4ada-b3d1-842e234be770"). InnerVolumeSpecName "kube-api-access-knwqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 20:33:38.973532 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:38.973441    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7705d7ef-f85a-4ada-b3d1-842e234be770-maas-api-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:33:38.973532 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:38.973471    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7705d7ef-f85a-4ada-b3d1-842e234be770-openshift-service-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:33:38.973532 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:38.973482    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knwqr\" (UniqueName: \"kubernetes.io/projected/7705d7ef-f85a-4ada-b3d1-842e234be770-kube-api-access-knwqr\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:33:39.283771 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:39.283737    2560 generic.go:356] "Generic (PLEG): container finished" podID="7705d7ef-f85a-4ada-b3d1-842e234be770" containerID="591b1bc8647c4e2be7f6b5cad163aa856054f7985734ddbac7cdba9f99a8e415" exitCode=0
Jul 14 20:33:39.283928 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:39.283779    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj"
Jul 14 20:33:39.283928 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:39.283816    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj" event={"ID":"7705d7ef-f85a-4ada-b3d1-842e234be770","Type":"ContainerDied","Data":"591b1bc8647c4e2be7f6b5cad163aa856054f7985734ddbac7cdba9f99a8e415"}
Jul 14 20:33:39.283928 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:39.283862    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj" event={"ID":"7705d7ef-f85a-4ada-b3d1-842e234be770","Type":"ContainerDied","Data":"5d8e216898be2e6dfc4e7b19c270ed4102d6bf0d95bcbdd06ceabbcee97d2c2b"}
Jul 14 20:33:39.283928 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:39.283884    2560 scope.go:117] "RemoveContainer" containerID="591b1bc8647c4e2be7f6b5cad163aa856054f7985734ddbac7cdba9f99a8e415"
Jul 14 20:33:39.293090 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:39.293074    2560 scope.go:117] "RemoveContainer" containerID="591b1bc8647c4e2be7f6b5cad163aa856054f7985734ddbac7cdba9f99a8e415"
Jul 14 20:33:39.293427 ip-10-0-133-168 kubenswrapper[2560]: E0714 20:33:39.293408    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"591b1bc8647c4e2be7f6b5cad163aa856054f7985734ddbac7cdba9f99a8e415\": container with ID starting with 591b1bc8647c4e2be7f6b5cad163aa856054f7985734ddbac7cdba9f99a8e415 not found: ID does not exist" containerID="591b1bc8647c4e2be7f6b5cad163aa856054f7985734ddbac7cdba9f99a8e415"
Jul 14 20:33:39.293479 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:39.293435    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"591b1bc8647c4e2be7f6b5cad163aa856054f7985734ddbac7cdba9f99a8e415"} err="failed to get container status \"591b1bc8647c4e2be7f6b5cad163aa856054f7985734ddbac7cdba9f99a8e415\": rpc error: code = NotFound desc = could not find container \"591b1bc8647c4e2be7f6b5cad163aa856054f7985734ddbac7cdba9f99a8e415\": container with ID starting with 591b1bc8647c4e2be7f6b5cad163aa856054f7985734ddbac7cdba9f99a8e415 not found: ID does not exist"
Jul 14 20:33:39.316967 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:39.316938    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj"]
Jul 14 20:33:39.329636 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:39.329613    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-10ef4d-c76ccbb94-2kcgj"]
Jul 14 20:33:39.889987 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:39.889952    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7705d7ef-f85a-4ada-b3d1-842e234be770" path="/var/lib/kubelet/pods/7705d7ef-f85a-4ada-b3d1-842e234be770/volumes"
Jul 14 20:33:53.791850 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:53.791813    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d04b66-5rs77"]
Jul 14 20:33:53.792322 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:53.792253    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fbbbedf-4fb9-4f2f-8ba7-508b12a620ea" containerName="test-curl-7704-140546066851568"
Jul 14 20:33:53.792322 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:53.792269    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fbbbedf-4fb9-4f2f-8ba7-508b12a620ea" containerName="test-curl-7704-140546066851568"
Jul 14 20:33:53.792322 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:53.792279    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7705d7ef-f85a-4ada-b3d1-842e234be770" containerName="maas-api"
Jul 14 20:33:53.792322 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:53.792285    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7705d7ef-f85a-4ada-b3d1-842e234be770" containerName="maas-api"
Jul 14 20:33:53.792458 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:53.792361    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7705d7ef-f85a-4ada-b3d1-842e234be770" containerName="maas-api"
Jul 14 20:33:53.792458 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:53.792372    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fbbbedf-4fb9-4f2f-8ba7-508b12a620ea" containerName="test-curl-7704-140546066851568"
Jul 14 20:33:53.795752 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:53.795688    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d04b66-5rs77"
Jul 14 20:33:53.797932 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:53.797901    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-56w6s\""
Jul 14 20:33:53.802031 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:53.802005    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d04b66-5rs77"]
Jul 14 20:33:53.910935 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:53.910898    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22cbe1c5-a9b4-4675-a905-d6c80532f177-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-d04b66-5rs77\" (UID: \"22cbe1c5-a9b4-4675-a905-d6c80532f177\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d04b66-5rs77"
Jul 14 20:33:54.011971 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:54.011918    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22cbe1c5-a9b4-4675-a905-d6c80532f177-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-d04b66-5rs77\" (UID: \"22cbe1c5-a9b4-4675-a905-d6c80532f177\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d04b66-5rs77"
Jul 14 20:33:54.012708 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:54.012685    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22cbe1c5-a9b4-4675-a905-d6c80532f177-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-d04b66-5rs77\" (UID: \"22cbe1c5-a9b4-4675-a905-d6c80532f177\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d04b66-5rs77"
Jul 14 20:33:54.108517 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:54.108429    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d04b66-5rs77"
Jul 14 20:33:54.245528 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:54.243051    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d04b66-5rs77"]
Jul 14 20:33:54.246510 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:33:54.246471    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22cbe1c5_a9b4_4675_a905_d6c80532f177.slice/crio-dd27c128a4c90961bc1b21063ab487e60dd8aba56b0025fc6a96364b0661e004 WatchSource:0}: Error finding container dd27c128a4c90961bc1b21063ab487e60dd8aba56b0025fc6a96364b0661e004: Status 404 returned error can't find the container with id dd27c128a4c90961bc1b21063ab487e60dd8aba56b0025fc6a96364b0661e004
Jul 14 20:33:54.360449 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:54.360360    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d04b66-5rs77" event={"ID":"22cbe1c5-a9b4-4675-a905-d6c80532f177","Type":"ContainerStarted","Data":"dd27c128a4c90961bc1b21063ab487e60dd8aba56b0025fc6a96364b0661e004"}
Jul 14 20:33:55.366842 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:55.366806    2560 generic.go:356] "Generic (PLEG): container finished" podID="22cbe1c5-a9b4-4675-a905-d6c80532f177" containerID="a43e0e37e48e5aeb49fbc8c116618022329cc778fe3105bc94080d68b8e0d6a5" exitCode=0
Jul 14 20:33:55.367247 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:55.366896    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d04b66-5rs77" event={"ID":"22cbe1c5-a9b4-4675-a905-d6c80532f177","Type":"ContainerDied","Data":"a43e0e37e48e5aeb49fbc8c116618022329cc778fe3105bc94080d68b8e0d6a5"}
Jul 14 20:33:56.506941 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:56.506916    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d04b66-5rs77"
Jul 14 20:33:56.534048 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:56.534023    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22cbe1c5-a9b4-4675-a905-d6c80532f177-maas-api-service-ca\") pod \"22cbe1c5-a9b4-4675-a905-d6c80532f177\" (UID: \"22cbe1c5-a9b4-4675-a905-d6c80532f177\") "
Jul 14 20:33:56.541004 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:56.534623    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22cbe1c5-a9b4-4675-a905-d6c80532f177-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "22cbe1c5-a9b4-4675-a905-d6c80532f177" (UID: "22cbe1c5-a9b4-4675-a905-d6c80532f177"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 20:33:56.635303 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:56.635218    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22cbe1c5-a9b4-4675-a905-d6c80532f177-maas-api-service-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:33:57.378402 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:57.378366    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d04b66-5rs77" event={"ID":"22cbe1c5-a9b4-4675-a905-d6c80532f177","Type":"ContainerDied","Data":"dd27c128a4c90961bc1b21063ab487e60dd8aba56b0025fc6a96364b0661e004"}
Jul 14 20:33:57.378579 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:57.378409    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d04b66-5rs77"
Jul 14 20:33:57.378579 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:33:57.378418    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd27c128a4c90961bc1b21063ab487e60dd8aba56b0025fc6a96364b0661e004"
Jul 14 20:34:04.280778 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:04.280745    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz"]
Jul 14 20:34:04.283417 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:04.281215    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz" podUID="433df555-f927-4267-aaeb-fb74aaa17f8a" containerName="maas-api" containerID="cri-o://3dfb27989cd828c746d300bd3ee8eff7564a6087ce2518cb7f61f91ad6487c91" gracePeriod=30
Jul 14 20:34:04.416986 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:04.416957    2560 generic.go:356] "Generic (PLEG): container finished" podID="433df555-f927-4267-aaeb-fb74aaa17f8a" containerID="3dfb27989cd828c746d300bd3ee8eff7564a6087ce2518cb7f61f91ad6487c91" exitCode=0
Jul 14 20:34:04.417134 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:04.417009    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz" event={"ID":"433df555-f927-4267-aaeb-fb74aaa17f8a","Type":"ContainerDied","Data":"3dfb27989cd828c746d300bd3ee8eff7564a6087ce2518cb7f61f91ad6487c91"}
Jul 14 20:34:04.535410 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:04.535351    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz"
Jul 14 20:34:04.592901 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:04.592863    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/433df555-f927-4267-aaeb-fb74aaa17f8a-maas-api-tls\") pod \"433df555-f927-4267-aaeb-fb74aaa17f8a\" (UID: \"433df555-f927-4267-aaeb-fb74aaa17f8a\") "
Jul 14 20:34:04.593093 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:04.592961    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/433df555-f927-4267-aaeb-fb74aaa17f8a-openshift-service-ca\") pod \"433df555-f927-4267-aaeb-fb74aaa17f8a\" (UID: \"433df555-f927-4267-aaeb-fb74aaa17f8a\") "
Jul 14 20:34:04.593093 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:04.592988    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pfvq\" (UniqueName: \"kubernetes.io/projected/433df555-f927-4267-aaeb-fb74aaa17f8a-kube-api-access-8pfvq\") pod \"433df555-f927-4267-aaeb-fb74aaa17f8a\" (UID: \"433df555-f927-4267-aaeb-fb74aaa17f8a\") "
Jul 14 20:34:04.593379 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:04.593356    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/433df555-f927-4267-aaeb-fb74aaa17f8a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "433df555-f927-4267-aaeb-fb74aaa17f8a" (UID: "433df555-f927-4267-aaeb-fb74aaa17f8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 20:34:04.595074 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:04.595047    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/433df555-f927-4267-aaeb-fb74aaa17f8a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "433df555-f927-4267-aaeb-fb74aaa17f8a" (UID: "433df555-f927-4267-aaeb-fb74aaa17f8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 20:34:04.595149 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:04.595116    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433df555-f927-4267-aaeb-fb74aaa17f8a-kube-api-access-8pfvq" (OuterVolumeSpecName: "kube-api-access-8pfvq") pod "433df555-f927-4267-aaeb-fb74aaa17f8a" (UID: "433df555-f927-4267-aaeb-fb74aaa17f8a"). InnerVolumeSpecName "kube-api-access-8pfvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 20:34:04.693622 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:04.693595    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/433df555-f927-4267-aaeb-fb74aaa17f8a-maas-api-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:34:04.693622 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:04.693620    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/433df555-f927-4267-aaeb-fb74aaa17f8a-openshift-service-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:34:04.693622 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:04.693629    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pfvq\" (UniqueName: \"kubernetes.io/projected/433df555-f927-4267-aaeb-fb74aaa17f8a-kube-api-access-8pfvq\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jul 14 20:34:05.423291 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:05.423259    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz"
Jul 14 20:34:05.423728 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:05.423293    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz" event={"ID":"433df555-f927-4267-aaeb-fb74aaa17f8a","Type":"ContainerDied","Data":"356a52dfdb10f5c74604202fad23c2dbdfaa8b15d749d03f43324df78c49d552"}
Jul 14 20:34:05.423728 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:05.423348    2560 scope.go:117] "RemoveContainer" containerID="3dfb27989cd828c746d300bd3ee8eff7564a6087ce2518cb7f61f91ad6487c91"
Jul 14 20:34:05.445038 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:05.445012    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz"]
Jul 14 20:34:05.451465 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:05.451442    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d04b66-7879bb66d9-g9fcz"]
Jul 14 20:34:05.889477 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:05.889445    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="433df555-f927-4267-aaeb-fb74aaa17f8a" path="/var/lib/kubelet/pods/433df555-f927-4267-aaeb-fb74aaa17f8a/volumes"
Jul 14 20:34:25.654302 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:25.654266    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-pqs6p_fccfa7ec-d7e0-4279-8824-94d22fd84ff2/manager/0.log"
Jul 14 20:34:25.796821 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:25.796789    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b96486789-tj8km_838f0bc8-3271-4987-abc1-76141aea6162/manager/0.log"
Jul 14 20:34:25.921764 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:25.921686    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-jfck4_d266cde7-b5bd-4ae6-8977-be6bed9824aa/server/0.log"
Jul 14 20:34:26.185630 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:26.185551    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c597f769-dckpj_8a4667e6-2144-4b5f-9b6a-4e07d9b75350/copy-manifests/0.log"
Jul 14 20:34:26.204740 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:26.204712    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c597f769-dckpj_8a4667e6-2144-4b5f-9b6a-4e07d9b75350/manager/0.log"
Jul 14 20:34:26.877904 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:26.877873    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7c45c66c84-h28pw_ed092f93-6175-400c-b4a2-6ece6b791610/maas-api/0.log"
Jul 14 20:34:26.994921 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:26.994890    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29734320-59n9c_67ef60bd-641a-4e12-b9a5-ee3b819a19fe/cleanup/0.log"
Jul 14 20:34:27.114669 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:27.114640    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29734335-fkdrw_1206f397-6723-4384-8e79-304508cbce93/cleanup/0.log"
Jul 14 20:34:27.353790 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:27.353762    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-84a35a9d-d9swv_fdc85368-987f-4c9f-986e-85b0f0b283fb/revoke-keys/0.log"
Jul 14 20:34:27.474864 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:27.474833    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-96fd5154-png4z_fc362cbd-667e-4ea4-b40b-7d46034c607a/revoke-keys/0.log"
Jul 14 20:34:27.721720 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:27.721647    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-0f00f6-p2msv_056d89e4-a506-47a8-bd63-43d4cec3c17a/revoke-keys/0.log"
Jul 14 20:34:27.844771 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:27.844744    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-f02785bb-xlwsk_92654054-1bfa-4df2-bf8a-e70920c1cd5f/revoke-keys/0.log"
Jul 14 20:34:28.191570 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:28.191525    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-9ead5203-9sdrj_cac4df9d-fe96-4dfa-87dc-7d36588dda49/revoke-keys/0.log"
Jul 14 20:34:28.310350 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:28.310325    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-407eaab7-4tfp2_f8e55e36-5c7d-4b54-934d-a04fd241df72/revoke-keys/0.log"
Jul 14 20:34:28.666314 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:28.666285    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-d04b66-5rs77_22cbe1c5-a9b4-4675-a905-d6c80532f177/revoke-keys/0.log"
Jul 14 20:34:30.559660 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:30.559629    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-8d7dq_c0c2a600-5f9a-4d03-aabf-d28814e14cad/kuadrant-console-plugin/0.log"
Jul 14 20:34:31.704994 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:31.704964    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d8f87cd4c-769pv_05036eb5-3926-42c7-916e-b631b97a599e/kube-auth-proxy/0.log"
Jul 14 20:34:32.102718 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:32.102692    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-57755_aafb96c1-4030-49f7-8927-3ef681049bf0/payload-pre-processing/0.log"
Jul 14 20:34:32.234777 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:32.234747    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-rwkrz_5799959e-dffa-4835-94fb-945fbeae2967/payload-processing/0.log"
Jul 14 20:34:32.856289 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:32.856257    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-k6mhb_7ecff220-68c4-4ac9-a7a5-727e9bccfabf/main/0.log"
Jul 14 20:34:33.100826 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:33.100793    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbz4x7v_a0832b36-4701-4c38-85de-25654036ce78/main/0.log"
Jul 14 20:34:33.346239 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:33.346207    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-dq25g_99a351dd-0cdd-475f-9909-9a6613594aa6/main/0.log"
Jul 14 20:34:40.217242 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:40.217201    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pp59q_120711d8-0534-470b-ab0f-5433b06fef47/global-pull-secret-syncer/0.log"
Jul 14 20:34:40.349004 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:40.348968    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xln8v_6c498088-c01f-4ddd-9c52-0972d109f662/konnectivity-agent/0.log"
Jul 14 20:34:40.430309 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:40.430274    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-168.ec2.internal_0536b894286084f9faaec29be192da95/haproxy/0.log"
Jul 14 20:34:45.773611 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:45.773580    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-8d7dq_c0c2a600-5f9a-4d03-aabf-d28814e14cad/kuadrant-console-plugin/0.log"
Jul 14 20:34:47.933247 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:47.933216    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_889dff78-34a1-4966-8973-70bd68aa050c/alertmanager/0.log"
Jul 14 20:34:47.961538 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:47.961505    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_889dff78-34a1-4966-8973-70bd68aa050c/config-reloader/0.log"
Jul 14 20:34:47.983851 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:47.983821    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_889dff78-34a1-4966-8973-70bd68aa050c/kube-rbac-proxy-web/0.log"
Jul 14 20:34:48.005607 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:48.005581    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_889dff78-34a1-4966-8973-70bd68aa050c/kube-rbac-proxy/0.log"
Jul 14 20:34:48.031164 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:48.031118    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_889dff78-34a1-4966-8973-70bd68aa050c/kube-rbac-proxy-metric/0.log"
Jul 14 20:34:48.054754 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:48.054725    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_889dff78-34a1-4966-8973-70bd68aa050c/prom-label-proxy/0.log"
Jul 14 20:34:48.077400 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:48.077371    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_889dff78-34a1-4966-8973-70bd68aa050c/init-config-reloader/0.log"
Jul 14 20:34:48.399329 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:48.399301    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gnrjv_f919f1fa-9d2a-4698-b875-63205715fa3c/node-exporter/0.log"
Jul 14 20:34:48.419646 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:48.419622    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gnrjv_f919f1fa-9d2a-4698-b875-63205715fa3c/kube-rbac-proxy/0.log"
Jul 14 20:34:48.442162 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:48.442143    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gnrjv_f919f1fa-9d2a-4698-b875-63205715fa3c/init-textfile/0.log"
Jul 14 20:34:48.542343 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:48.542286    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-624qf_dc4584b4-2397-4ec4-9cf4-8f2edfa2151f/kube-rbac-proxy-main/0.log"
Jul 14 20:34:48.563650 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:48.563620    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-624qf_dc4584b4-2397-4ec4-9cf4-8f2edfa2151f/kube-rbac-proxy-self/0.log"
Jul 14 20:34:48.586663 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:48.586640    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-624qf_dc4584b4-2397-4ec4-9cf4-8f2edfa2151f/openshift-state-metrics/0.log"
Jul 14 20:34:48.622298 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:48.622262    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6d8962ff-61b2-4a31-a784-dfe49041f886/prometheus/0.log"
Jul 14 20:34:48.643881 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:48.643857    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6d8962ff-61b2-4a31-a784-dfe49041f886/config-reloader/0.log"
Jul 14 20:34:48.665600 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:48.665532    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6d8962ff-61b2-4a31-a784-dfe49041f886/thanos-sidecar/0.log"
Jul 14 20:34:48.686732 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:48.686714    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6d8962ff-61b2-4a31-a784-dfe49041f886/kube-rbac-proxy-web/0.log"
Jul 14 20:34:48.708064 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:48.708041    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6d8962ff-61b2-4a31-a784-dfe49041f886/kube-rbac-proxy/0.log"
Jul 14 20:34:48.731120 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:48.731093    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6d8962ff-61b2-4a31-a784-dfe49041f886/kube-rbac-proxy-thanos/0.log"
Jul 14 20:34:48.752946 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:48.752925    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6d8962ff-61b2-4a31-a784-dfe49041f886/init-config-reloader/0.log"
Jul 14 20:34:48.945425 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:48.945336    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d79d9b6d5-frn87_8caccfe2-d79e-4bf5-b913-e848ba5e9275/thanos-query/0.log"
Jul 14 20:34:48.966952 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:48.966922    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d79d9b6d5-frn87_8caccfe2-d79e-4bf5-b913-e848ba5e9275/kube-rbac-proxy-web/0.log"
Jul 14 20:34:48.990611 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:48.990587    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d79d9b6d5-frn87_8caccfe2-d79e-4bf5-b913-e848ba5e9275/kube-rbac-proxy/0.log"
Jul 14 20:34:49.010529 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:49.010508    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d79d9b6d5-frn87_8caccfe2-d79e-4bf5-b913-e848ba5e9275/prom-label-proxy/0.log"
Jul 14 20:34:49.031658 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:49.031635    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d79d9b6d5-frn87_8caccfe2-d79e-4bf5-b913-e848ba5e9275/kube-rbac-proxy-rules/0.log"
Jul 14 20:34:49.054826 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:49.054797    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d79d9b6d5-frn87_8caccfe2-d79e-4bf5-b913-e848ba5e9275/kube-rbac-proxy-metrics/0.log"
Jul 14 20:34:50.521374 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.521337    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tnh4v/perf-node-gather-daemonset-jdg9r"]
Jul 14 20:34:50.521752 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.521700    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="433df555-f927-4267-aaeb-fb74aaa17f8a" containerName="maas-api"
Jul 14 20:34:50.521752 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.521710    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="433df555-f927-4267-aaeb-fb74aaa17f8a" containerName="maas-api"
Jul 14 20:34:50.521752 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.521721    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22cbe1c5-a9b4-4675-a905-d6c80532f177" containerName="revoke-keys"
Jul 14 20:34:50.521752 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.521727    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="22cbe1c5-a9b4-4675-a905-d6c80532f177" containerName="revoke-keys"
Jul 14 20:34:50.521883 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.521793    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="22cbe1c5-a9b4-4675-a905-d6c80532f177" containerName="revoke-keys"
Jul 14 20:34:50.521883 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.521805    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="433df555-f927-4267-aaeb-fb74aaa17f8a" containerName="maas-api"
Jul 14 20:34:50.524773 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.524757    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-jdg9r"
Jul 14 20:34:50.526871 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.526854    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tnh4v\"/\"default-dockercfg-7bx5v\""
Jul 14 20:34:50.526967 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.526899    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tnh4v\"/\"openshift-service-ca.crt\""
Jul 14 20:34:50.527790 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.527772    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tnh4v\"/\"kube-root-ca.crt\""
Jul 14 20:34:50.535054 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.534966    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tnh4v/perf-node-gather-daemonset-jdg9r"]
Jul 14 20:34:50.699387 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.699358    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/90a956bc-16e4-4a7d-94c0-68535f76cb67-proc\") pod \"perf-node-gather-daemonset-jdg9r\" (UID: \"90a956bc-16e4-4a7d-94c0-68535f76cb67\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-jdg9r"
Jul 14 20:34:50.699533 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.699435    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90a956bc-16e4-4a7d-94c0-68535f76cb67-lib-modules\") pod \"perf-node-gather-daemonset-jdg9r\" (UID: \"90a956bc-16e4-4a7d-94c0-68535f76cb67\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-jdg9r"
Jul 14 20:34:50.699533 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.699455    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/90a956bc-16e4-4a7d-94c0-68535f76cb67-podres\") pod \"perf-node-gather-daemonset-jdg9r\" (UID: \"90a956bc-16e4-4a7d-94c0-68535f76cb67\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-jdg9r"
Jul 14 20:34:50.699533 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.699470    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc6t9\" (UniqueName: \"kubernetes.io/projected/90a956bc-16e4-4a7d-94c0-68535f76cb67-kube-api-access-sc6t9\") pod \"perf-node-gather-daemonset-jdg9r\" (UID: \"90a956bc-16e4-4a7d-94c0-68535f76cb67\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-jdg9r"
Jul 14 20:34:50.699533 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.699485    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90a956bc-16e4-4a7d-94c0-68535f76cb67-sys\") pod \"perf-node-gather-daemonset-jdg9r\" (UID: \"90a956bc-16e4-4a7d-94c0-68535f76cb67\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-jdg9r"
Jul 14 20:34:50.800783 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.800753    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90a956bc-16e4-4a7d-94c0-68535f76cb67-lib-modules\") pod \"perf-node-gather-daemonset-jdg9r\" (UID: \"90a956bc-16e4-4a7d-94c0-68535f76cb67\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-jdg9r"
Jul 14 20:34:50.800957 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.800788    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/90a956bc-16e4-4a7d-94c0-68535f76cb67-podres\") pod \"perf-node-gather-daemonset-jdg9r\" (UID: \"90a956bc-16e4-4a7d-94c0-68535f76cb67\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-jdg9r"
Jul 14 20:34:50.800957 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.800808    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc6t9\" (UniqueName: \"kubernetes.io/projected/90a956bc-16e4-4a7d-94c0-68535f76cb67-kube-api-access-sc6t9\") pod \"perf-node-gather-daemonset-jdg9r\" (UID: \"90a956bc-16e4-4a7d-94c0-68535f76cb67\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-jdg9r"
Jul 14 20:34:50.800957 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.800826    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90a956bc-16e4-4a7d-94c0-68535f76cb67-sys\") pod \"perf-node-gather-daemonset-jdg9r\" (UID: \"90a956bc-16e4-4a7d-94c0-68535f76cb67\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-jdg9r"
Jul 14 20:34:50.800957 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.800895    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/90a956bc-16e4-4a7d-94c0-68535f76cb67-proc\") pod \"perf-node-gather-daemonset-jdg9r\" (UID: \"90a956bc-16e4-4a7d-94c0-68535f76cb67\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-jdg9r"
Jul 14 20:34:50.800957 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.800947    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90a956bc-16e4-4a7d-94c0-68535f76cb67-lib-modules\") pod \"perf-node-gather-daemonset-jdg9r\" (UID: \"90a956bc-16e4-4a7d-94c0-68535f76cb67\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-jdg9r"
Jul 14 20:34:50.801199 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.800971    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90a956bc-16e4-4a7d-94c0-68535f76cb67-sys\") pod \"perf-node-gather-daemonset-jdg9r\" (UID: \"90a956bc-16e4-4a7d-94c0-68535f76cb67\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-jdg9r"
Jul 14 20:34:50.801199 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.800947    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/90a956bc-16e4-4a7d-94c0-68535f76cb67-podres\") pod \"perf-node-gather-daemonset-jdg9r\" (UID: \"90a956bc-16e4-4a7d-94c0-68535f76cb67\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-jdg9r"
Jul 14 20:34:50.801199 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.801000    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/90a956bc-16e4-4a7d-94c0-68535f76cb67-proc\") pod \"perf-node-gather-daemonset-jdg9r\" (UID: \"90a956bc-16e4-4a7d-94c0-68535f76cb67\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-jdg9r"
Jul 14 20:34:50.809637 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.809613    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc6t9\" (UniqueName: \"kubernetes.io/projected/90a956bc-16e4-4a7d-94c0-68535f76cb67-kube-api-access-sc6t9\") pod \"perf-node-gather-daemonset-jdg9r\" (UID: \"90a956bc-16e4-4a7d-94c0-68535f76cb67\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-jdg9r"
Jul 14 20:34:50.835815 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.835785    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-jdg9r"
Jul 14 20:34:50.972917 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:50.972865    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tnh4v/perf-node-gather-daemonset-jdg9r"]
Jul 14 20:34:50.975012 ip-10-0-133-168 kubenswrapper[2560]: W0714 20:34:50.974985    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod90a956bc_16e4_4a7d_94c0_68535f76cb67.slice/crio-c8f21d5d6dd1502de8a52aaa625c40af16ceed6d8d8bd17ed0c56ad55797ca4f WatchSource:0}: Error finding container c8f21d5d6dd1502de8a52aaa625c40af16ceed6d8d8bd17ed0c56ad55797ca4f: Status 404 returned error can't find the container with id c8f21d5d6dd1502de8a52aaa625c40af16ceed6d8d8bd17ed0c56ad55797ca4f
Jul 14 20:34:51.662762 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:51.662725    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-jdg9r" event={"ID":"90a956bc-16e4-4a7d-94c0-68535f76cb67","Type":"ContainerStarted","Data":"161c83b8a9eff4aa0703f13fdf1dddd6c13ace001e692717ba28394063191013"}
Jul 14 20:34:51.663154 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:51.662807    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-jdg9r"
Jul 14 20:34:51.663154 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:51.662823    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-jdg9r" event={"ID":"90a956bc-16e4-4a7d-94c0-68535f76cb67","Type":"ContainerStarted","Data":"c8f21d5d6dd1502de8a52aaa625c40af16ceed6d8d8bd17ed0c56ad55797ca4f"}
Jul 14 20:34:51.681253 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:51.681209    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-jdg9r" podStartSLOduration=1.6811939740000001 podStartE2EDuration="1.681193974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 20:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:34:51.679111098 +0000 UTC m=+2844.279414793" watchObservedRunningTime="2026-07-14 20:34:51.681193974 +0000 UTC m=+2844.281497676"
Jul 14 20:34:53.044246 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:53.044220    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mgqdj_8fafc082-f569-4875-8d12-db7504865e6d/dns/0.log"
Jul 14 20:34:53.068684 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:53.068653    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mgqdj_8fafc082-f569-4875-8d12-db7504865e6d/kube-rbac-proxy/0.log"
Jul 14 20:34:53.146647 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:53.146610    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bz7rb_ced44ef1-063d-48b9-81bc-412dd2a95f74/dns-node-resolver/0.log"
Jul 14 20:34:53.841626 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:53.841599    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qwdn7_1b5468c6-fd78-4acc-b882-cf389722a101/node-ca/0.log"
Jul 14 20:34:55.002927 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:55.002895    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d8f87cd4c-769pv_05036eb5-3926-42c7-916e-b631b97a599e/kube-auth-proxy/0.log"
Jul 14 20:34:55.208083 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:55.208057    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-57755_aafb96c1-4030-49f7-8927-3ef681049bf0/payload-pre-processing/0.log"
Jul 14 20:34:55.267681 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:55.267594    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-rwkrz_5799959e-dffa-4835-94fb-945fbeae2967/payload-processing/0.log"
Jul 14 20:34:56.035714 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:56.035687    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fd6mw_86c2f79a-dd6d-46b1-ac5c-e1d00a7a652b/serve-healthcheck-canary/0.log"
Jul 14 20:34:56.606498 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:56.606465    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d2d6k_1fd75dce-40e5-494c-92c4-eac34f1fc302/kube-rbac-proxy/0.log"
Jul 14 20:34:56.629850 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:56.629822    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d2d6k_1fd75dce-40e5-494c-92c4-eac34f1fc302/exporter/0.log"
Jul 14 20:34:56.662058 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:56.662030    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d2d6k_1fd75dce-40e5-494c-92c4-eac34f1fc302/extractor/0.log"
Jul 14 20:34:57.680830 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:57.680805    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-jdg9r"
Jul 14 20:34:59.144998 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:59.144962    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-pqs6p_fccfa7ec-d7e0-4279-8824-94d22fd84ff2/manager/0.log"
Jul 14 20:34:59.316705 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:59.316668    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b96486789-tj8km_838f0bc8-3271-4987-abc1-76141aea6162/manager/0.log"
Jul 14 20:34:59.348368 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:59.348343    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-jfck4_d266cde7-b5bd-4ae6-8977-be6bed9824aa/server/0.log"
Jul 14 20:34:59.579317 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:59.579237    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c597f769-dckpj_8a4667e6-2144-4b5f-9b6a-4e07d9b75350/manager/0.log"
Jul 14 20:34:59.599936 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:34:59.599915    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c597f769-dckpj_8a4667e6-2144-4b5f-9b6a-4e07d9b75350/copy-manifests/0.log"
Jul 14 20:35:00.696952 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:35:00.696915    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d8f87cd4c-769pv_05036eb5-3926-42c7-916e-b631b97a599e/kube-auth-proxy/0.log"
Jul 14 20:35:00.839033 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:35:00.838936    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-57755_aafb96c1-4030-49f7-8927-3ef681049bf0/payload-pre-processing/0.log"
Jul 14 20:35:00.866289 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:35:00.866256    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-rwkrz_5799959e-dffa-4835-94fb-945fbeae2967/payload-processing/0.log"
Jul 14 20:35:01.135351 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:35:01.135254    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-77d6d59f96-69z6h_411bcb53-f540-49a8-8665-3b257034d8f4/manager/0.log"
Jul 14 20:35:02.259150 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:35:02.259125    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d8f87cd4c-769pv_05036eb5-3926-42c7-916e-b631b97a599e/kube-auth-proxy/0.log"
Jul 14 20:35:02.408780 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:35:02.408745    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-57755_aafb96c1-4030-49f7-8927-3ef681049bf0/payload-pre-processing/0.log"
Jul 14 20:35:02.441701 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:35:02.441572    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-rwkrz_5799959e-dffa-4835-94fb-945fbeae2967/payload-processing/0.log"
Jul 14 20:35:07.822628 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:35:07.822600    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-smrzh_b5bfcb6a-4113-4273-a3e2-d6b64f2e4937/kube-multus-additional-cni-plugins/0.log"
Jul 14 20:35:07.846056 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:35:07.846030    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-smrzh_b5bfcb6a-4113-4273-a3e2-d6b64f2e4937/egress-router-binary-copy/0.log"
Jul 14 20:35:07.871880 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:35:07.871851    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-smrzh_b5bfcb6a-4113-4273-a3e2-d6b64f2e4937/cni-plugins/0.log"
Jul 14 20:35:07.893668 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:35:07.893641    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-smrzh_b5bfcb6a-4113-4273-a3e2-d6b64f2e4937/bond-cni-plugin/0.log"
Jul 14 20:35:07.918260 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:35:07.918234    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-smrzh_b5bfcb6a-4113-4273-a3e2-d6b64f2e4937/routeoverride-cni/0.log"
Jul 14 20:35:07.948232 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:35:07.948194    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-smrzh_b5bfcb6a-4113-4273-a3e2-d6b64f2e4937/whereabouts-cni-bincopy/0.log"
Jul 14 20:35:07.975191 ip-10-0-133-168 kubenswrapper[2560]: I0714 20:35:07.975144    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-smrzh_b5bfcb6a-4113-4273-a3e2-d6b64f2e4937/whereabouts-cni/0.log"