Jul 27 18:28:59.441437 ip-10-0-128-196 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 18:28:59.859554 ip-10-0-128-196 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 18:28:59.859554 ip-10-0-128-196 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 18:28:59.859554 ip-10-0-128-196 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 18:28:59.859554 ip-10-0-128-196 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 18:28:59.859554 ip-10-0-128-196 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 18:28:59.860594 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.860506    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 18:28:59.864102 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864086    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 18:28:59.864156 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864104    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 18:28:59.864156 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864108    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 18:28:59.864156 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864111    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 18:28:59.864156 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864127    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 18:28:59.864156 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864130    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 18:28:59.864156 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864135    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 18:28:59.864156 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864139    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 18:28:59.864156 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864142    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 18:28:59.864156 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864145    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 18:28:59.864156 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864147    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 18:28:59.864156 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864150    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 18:28:59.864156 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864152    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 18:28:59.864156 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864155    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 18:28:59.864156 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864157    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 18:28:59.864156 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864160    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 18:28:59.864156 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864163    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 18:28:59.864578 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864166    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 18:28:59.864578 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864173    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 18:28:59.864578 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864176    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 18:28:59.864578 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864179    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 18:28:59.864578 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864181    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 18:28:59.864578 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864183    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 18:28:59.864578 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864186    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 18:28:59.864578 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864188    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 18:28:59.864578 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864191    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 18:28:59.864578 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864193    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 18:28:59.864578 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864196    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 18:28:59.864578 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864199    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 18:28:59.864578 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864201    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 18:28:59.864578 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864204    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 18:28:59.864578 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864206    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 18:28:59.864578 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864209    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 18:28:59.864578 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864211    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 18:28:59.864578 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864214    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 18:28:59.864578 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864216    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 18:28:59.864578 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864218    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 18:28:59.865049 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864221    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 18:28:59.865049 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864223    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 18:28:59.865049 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864226    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 18:28:59.865049 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864228    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 18:28:59.865049 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864231    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 18:28:59.865049 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864233    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 18:28:59.865049 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864236    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 18:28:59.865049 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864238    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 18:28:59.865049 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864241    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 18:28:59.865049 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864243    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 18:28:59.865049 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864247    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 18:28:59.865049 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864250    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 18:28:59.865049 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864253    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 18:28:59.865049 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864256    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 18:28:59.865049 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864259    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 18:28:59.865049 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864261    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 18:28:59.865049 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864264    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 18:28:59.865049 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864266    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 18:28:59.865049 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864269    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 18:28:59.865049 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864271    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 18:28:59.865583 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864274    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 18:28:59.865583 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864276    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 18:28:59.865583 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864279    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 18:28:59.865583 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864282    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 18:28:59.865583 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864284    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 18:28:59.865583 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864286    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 18:28:59.865583 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864289    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 18:28:59.865583 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864292    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 18:28:59.865583 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864294    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 18:28:59.865583 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864297    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 18:28:59.865583 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864299    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 18:28:59.865583 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864302    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 18:28:59.865583 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864304    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 18:28:59.865583 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864307    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 18:28:59.865583 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864309    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 18:28:59.865583 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864312    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 18:28:59.865583 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864314    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 18:28:59.865583 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864317    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 18:28:59.865583 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864319    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 18:28:59.865583 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864322    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 18:28:59.866060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864324    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 18:28:59.866060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864327    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 18:28:59.866060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864329    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 18:28:59.866060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864332    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 18:28:59.866060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864335    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 18:28:59.866060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864338    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 18:28:59.866060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864343    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 18:28:59.866060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864346    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 18:28:59.866060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864349    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 18:28:59.866060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864352    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 18:28:59.866060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864354    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 18:28:59.866060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864358    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 18:28:59.866060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864360    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 18:28:59.866060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864363    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 18:28:59.866060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864365    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 18:28:59.866060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864368    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 18:28:59.866060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864370    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 18:28:59.866060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864372    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 18:28:59.866060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864375    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 18:28:59.866533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864379    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 18:28:59.866533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864382    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 18:28:59.866533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.864385    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 18:28:59.866533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865921    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 18:28:59.866533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865928    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 18:28:59.866533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865932    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 18:28:59.866533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865935    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 18:28:59.866533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865938    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 18:28:59.866533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865941    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 18:28:59.866533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865943    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 18:28:59.866533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865946    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 18:28:59.866533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865949    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 18:28:59.866533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865951    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 18:28:59.866533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865954    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 18:28:59.866533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865956    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 18:28:59.866533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865958    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 18:28:59.866533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865961    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 18:28:59.866533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865965    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 18:28:59.866533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865968    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 18:28:59.866533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865970    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 18:28:59.867060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865973    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 18:28:59.867060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865976    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 18:28:59.867060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865979    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 18:28:59.867060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865981    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 18:28:59.867060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865983    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 18:28:59.867060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865986    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 18:28:59.867060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865988    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 18:28:59.867060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865991    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 18:28:59.867060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865993    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 18:28:59.867060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865995    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 18:28:59.867060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.865998    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 18:28:59.867060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866000    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 18:28:59.867060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866003    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 18:28:59.867060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866005    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 18:28:59.867060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866008    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 18:28:59.867060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866010    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 18:28:59.867060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866013    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 18:28:59.867060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866015    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 18:28:59.867060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866018    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 18:28:59.867060 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866020    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 18:28:59.867568 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866023    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 18:28:59.867568 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866025    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 18:28:59.867568 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866028    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 18:28:59.867568 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866030    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 18:28:59.867568 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866032    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 18:28:59.867568 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866035    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 18:28:59.867568 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866038    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 18:28:59.867568 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866042    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 18:28:59.867568 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866046    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 18:28:59.867568 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866049    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 18:28:59.867568 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866053    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 18:28:59.867568 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866055    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 18:28:59.867568 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866058    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 18:28:59.867568 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866060    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 18:28:59.867568 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866064    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 18:28:59.867568 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866067    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 18:28:59.867568 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866070    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 18:28:59.867568 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866073    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 18:28:59.867568 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866076    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 18:28:59.867568 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866078    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 18:28:59.868062 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866081    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 18:28:59.868062 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866083    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 18:28:59.868062 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866086    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 18:28:59.868062 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866088    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 18:28:59.868062 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866091    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 18:28:59.868062 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866093    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 18:28:59.868062 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866096    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 18:28:59.868062 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866098    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 18:28:59.868062 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866101    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 18:28:59.868062 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866103    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 18:28:59.868062 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866106    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 18:28:59.868062 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866108    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 18:28:59.868062 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866111    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 18:28:59.868062 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866130    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 18:28:59.868062 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866135    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 18:28:59.868062 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866139    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 18:28:59.868062 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866142    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 18:28:59.868062 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866146    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 18:28:59.868062 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866149    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 18:28:59.868533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866152    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 18:28:59.868533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866156    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 18:28:59.868533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866161    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 18:28:59.868533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866165    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 18:28:59.868533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866169    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 18:28:59.868533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866172    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 18:28:59.868533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866175    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 18:28:59.868533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866178    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 18:28:59.868533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866182    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 18:28:59.868533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866185    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 18:28:59.868533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866188    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 18:28:59.868533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866191    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 18:28:59.868533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866194    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 18:28:59.868533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866197    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 18:28:59.868533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866199    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 18:28:59.868533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866202    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 18:28:59.868533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866204    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 18:28:59.868533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866208    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 18:28:59.868533 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866211    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 18:28:59.869012 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866214    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 18:28:59.869012 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866217    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 18:28:59.869012 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866220    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 18:28:59.869012 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866222    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 18:28:59.869012 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866598    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 18:28:59.869012 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866602    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 18:28:59.869012 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866606    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 18:28:59.869012 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866609    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 18:28:59.869012 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866612    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 18:28:59.869012 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866615    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 18:28:59.869012 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866619    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 18:28:59.869012 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866622    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 18:28:59.869012 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866625    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 18:28:59.869012 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866628    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 18:28:59.869012 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866631    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 18:28:59.869012 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866634    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 18:28:59.869012 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866637    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 18:28:59.869012 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866640    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 18:28:59.869012 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866643    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 18:28:59.869012 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866646    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 18:28:59.869618 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866648    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 18:28:59.869618 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866651    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 18:28:59.869618 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866654    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 18:28:59.869618 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866656    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 18:28:59.869618 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866659    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 18:28:59.869618 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866661    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 18:28:59.869618 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866664    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 18:28:59.869618 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866666    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 18:28:59.869618 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866669    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 18:28:59.869618 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866671    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 18:28:59.869618 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866673    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 18:28:59.869618 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866676    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 18:28:59.869618 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866678    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 18:28:59.869618 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866681    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 18:28:59.869618 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866683    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 18:28:59.869618 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866686    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 18:28:59.869618 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866689    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 18:28:59.869618 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866691    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 18:28:59.869618 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866694    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 18:28:59.869618 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866696    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 18:28:59.870098 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866698    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 18:28:59.870098 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866701    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 18:28:59.870098 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866703    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 18:28:59.870098 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866706    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 18:28:59.870098 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866708    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 18:28:59.870098 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866711    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 18:28:59.870098 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866713    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 18:28:59.870098 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866716    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 18:28:59.870098 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866718    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 18:28:59.870098 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866722    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 18:28:59.870098 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866726    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 18:28:59.870098 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866729    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 18:28:59.870098 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866732    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 18:28:59.870098 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866734    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 18:28:59.870098 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866737    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 18:28:59.870098 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866740    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 18:28:59.870098 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866742    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 18:28:59.870098 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866745    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 18:28:59.870098 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866747    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 18:28:59.870098 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866749    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 18:28:59.870595 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866752    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 18:28:59.870595 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866754    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 18:28:59.870595 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866757    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 18:28:59.870595 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866759    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 18:28:59.870595 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866762    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 18:28:59.870595 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866764    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 18:28:59.870595 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866766    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 18:28:59.870595 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866769    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 18:28:59.870595 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866772    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 18:28:59.870595 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866775    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 18:28:59.870595 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866777    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 18:28:59.870595 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866780    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 18:28:59.870595 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866783    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 18:28:59.870595 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866786    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 18:28:59.870595 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866788    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 18:28:59.870595 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866791    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 18:28:59.870595 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866793    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 18:28:59.870595 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866795    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 18:28:59.870595 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866798    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 18:28:59.870595 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866800    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 18:28:59.871084 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866803    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 18:28:59.871084 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866806    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 18:28:59.871084 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866809    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 18:28:59.871084 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866811    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 18:28:59.871084 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866814    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 18:28:59.871084 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866816    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 18:28:59.871084 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866819    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 18:28:59.871084 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866821    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 18:28:59.871084 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866824    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 18:28:59.871084 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866826    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 18:28:59.871084 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866829    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 18:28:59.871084 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866831    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 18:28:59.871084 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866833    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 18:28:59.871084 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866836    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 18:28:59.871084 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866838    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 18:28:59.871084 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866840    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 18:28:59.871084 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866843    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 18:28:59.871084 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866845    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 18:28:59.871084 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866848    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 18:28:59.871084 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866850    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 18:28:59.871577 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866852    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 18:28:59.871577 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866855    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 18:28:59.871577 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.866858    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 18:28:59.871577 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.866934    2566 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 18:28:59.871577 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.866945    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 18:28:59.871577 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.866952    2566 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 18:28:59.871577 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.866956    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 18:28:59.871577 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.866960    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 18:28:59.871577 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.866964    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 18:28:59.871577 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.866968    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 18:28:59.871577 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.866972    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 18:28:59.871577 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.866980    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 18:28:59.871577 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.866983    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 18:28:59.871577 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.866987    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 18:28:59.871577 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.866990    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 18:28:59.871577 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.866993    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 18:28:59.871577 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.866997    2566 flags.go:64] FLAG: --cgroup-root=""
Jul 27 18:28:59.871577 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867000    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 18:28:59.871577 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867003    2566 flags.go:64] FLAG: --client-ca-file=""
Jul 27 18:28:59.871577 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867006    2566 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 18:28:59.871577 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867009    2566 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 18:28:59.871577 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867013    2566 flags.go:64] FLAG: --cluster-domain=""
Jul 27 18:28:59.871577 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867016    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 18:28:59.871577 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867019    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 18:28:59.872163 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867022    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 18:28:59.872163 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867026    2566 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 18:28:59.872163 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867029    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 18:28:59.872163 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867032    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 18:28:59.872163 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867035    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 18:28:59.872163 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867038    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 18:28:59.872163 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867041    2566 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 18:28:59.872163 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867044    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 18:28:59.872163 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867047    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 18:28:59.872163 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867050    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 18:28:59.872163 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867053    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 18:28:59.872163 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867057    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 18:28:59.872163 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867060    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 18:28:59.872163 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867063    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 18:28:59.872163 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867066    2566 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 18:28:59.872163 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867069    2566 flags.go:64] FLAG: --enable-server="true"
Jul 27 18:28:59.872163 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867071    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 18:28:59.872163 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867076    2566 flags.go:64] FLAG: --event-burst="100"
Jul 27 18:28:59.872163 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867079    2566 flags.go:64] FLAG: --event-qps="50"
Jul 27 18:28:59.872163 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867084    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 18:28:59.872163 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867087    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 18:28:59.872163 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867090    2566 flags.go:64] FLAG: --eviction-hard=""
Jul 27 18:28:59.872163 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867093    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 18:28:59.872163 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867096    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 18:28:59.872780 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867099    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 18:28:59.872780 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867103    2566 flags.go:64] FLAG: --eviction-soft=""
Jul 27 18:28:59.872780 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867106    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 18:28:59.872780 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867109    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 18:28:59.872780 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867112    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 18:28:59.872780 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867128    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 18:28:59.872780 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867131    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 18:28:59.872780 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867134    2566 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 18:28:59.872780 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867137    2566 flags.go:64] FLAG: --feature-gates=""
Jul 27 18:28:59.872780 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867141    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 18:28:59.872780 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867144    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 18:28:59.872780 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867147    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 18:28:59.872780 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867150    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 18:28:59.872780 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867153    2566 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 18:28:59.872780 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867156    2566 flags.go:64] FLAG: --help="false"
Jul 27 18:28:59.872780 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867158    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-128-196.ec2.internal"
Jul 27 18:28:59.872780 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867161    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 18:28:59.872780 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867164    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 18:28:59.872780 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867167    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 18:28:59.872780 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867170    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 18:28:59.872780 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867173    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 18:28:59.872780 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867176    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 18:28:59.872780 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867179    2566 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 18:28:59.873422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867182    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 18:28:59.873422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867184    2566 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 18:28:59.873422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867187    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 18:28:59.873422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867190    2566 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 18:28:59.873422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867195    2566 flags.go:64] FLAG: --kube-reserved=""
Jul 27 18:28:59.873422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867197    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 18:28:59.873422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867200    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 18:28:59.873422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867203    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 18:28:59.873422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867206    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 18:28:59.873422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867209    2566 flags.go:64] FLAG: --lock-file=""
Jul 27 18:28:59.873422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867212    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 18:28:59.873422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867216    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 18:28:59.873422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867218    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 18:28:59.873422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867224    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 18:28:59.873422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867226    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 18:28:59.873422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867229    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 18:28:59.873422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867232    2566 flags.go:64] FLAG: --logging-format="text"
Jul 27 18:28:59.873422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867235    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 18:28:59.873422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867238    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 18:28:59.873422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867241    2566 flags.go:64] FLAG: --manifest-url=""
Jul 27 18:28:59.873422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867244    2566 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 18:28:59.873422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867248    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 18:28:59.873422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867251    2566 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 18:28:59.873422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867255    2566 flags.go:64] FLAG: --max-pods="110"
Jul 27 18:28:59.873422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867257    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 18:28:59.874040 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867260    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 18:28:59.874040 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867263    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 18:28:59.874040 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867266    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 18:28:59.874040 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867269    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 18:28:59.874040 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867272    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 18:28:59.874040 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867275    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 18:28:59.874040 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867286    2566 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 18:28:59.874040 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867288    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 18:28:59.874040 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867291    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 18:28:59.874040 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867294    2566 flags.go:64] FLAG: --pod-cidr=""
Jul 27 18:28:59.874040 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867297    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 18:28:59.874040 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867303    2566 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 18:28:59.874040 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867307    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 18:28:59.874040 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867310    2566 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 18:28:59.874040 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867313    2566 flags.go:64] FLAG: --port="10250"
Jul 27 18:28:59.874040 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867316    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 18:28:59.874040 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867319    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02bf24b0f3bb8e62e"
Jul 27 18:28:59.874040 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867322    2566 flags.go:64] FLAG: --qos-reserved=""
Jul 27 18:28:59.874040 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867325    2566 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 18:28:59.874040 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867328    2566 flags.go:64] FLAG: --register-node="true"
Jul 27 18:28:59.874040 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867331    2566 flags.go:64] FLAG: --register-with-taints=""
Jul 27 18:28:59.874040 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867334    2566 flags.go:64] FLAG: --registry-burst="10"
Jul 27 18:28:59.874040 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867337    2566 flags.go:64] FLAG: --registry-qps="5"
Jul 27 18:28:59.874040 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867340    2566 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 18:28:59.874675 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867343    2566 flags.go:64] FLAG: --reserved-memory=""
Jul 27 18:28:59.874675 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867347    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 18:28:59.874675 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867349    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 18:28:59.874675 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867352    2566 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 18:28:59.874675 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867355    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 18:28:59.874675 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867358    2566 flags.go:64] FLAG: --runonce="false"
Jul 27 18:28:59.874675 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867360    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 18:28:59.874675 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867363    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 18:28:59.874675 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867366    2566 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 18:28:59.874675 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867369    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 18:28:59.874675 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867372    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 18:28:59.874675 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867375    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 18:28:59.874675 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867378    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 18:28:59.874675 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867380    2566 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 18:28:59.874675 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867383    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 18:28:59.874675 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867386    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 18:28:59.874675 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867389    2566 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 18:28:59.874675 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867391    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 18:28:59.874675 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867394    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 18:28:59.874675 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867397    2566 flags.go:64] FLAG: --system-cgroups=""
Jul 27 18:28:59.874675 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867400    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 18:28:59.874675 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867405    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 18:28:59.874675 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867408    2566 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 18:28:59.874675 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867411    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 18:28:59.874675 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867415    2566 flags.go:64] FLAG: --tls-min-version=""
Jul 27 18:28:59.875302 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867418    2566 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 18:28:59.875302 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867421    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 18:28:59.875302 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867424    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 18:28:59.875302 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867427    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 18:28:59.875302 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867430    2566 flags.go:64] FLAG: --v="2"
Jul 27 18:28:59.875302 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867434    2566 flags.go:64] FLAG: --version="false"
Jul 27 18:28:59.875302 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867438    2566 flags.go:64] FLAG: --vmodule=""
Jul 27 18:28:59.875302 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867442    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 18:28:59.875302 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.867445    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 18:28:59.875302 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867557    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 18:28:59.875302 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867560    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 18:28:59.875302 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867563    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 18:28:59.875302 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867566    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 18:28:59.875302 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867568    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 18:28:59.875302 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867571    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 18:28:59.875302 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867574    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 18:28:59.875302 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867576    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 18:28:59.875302 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867579    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 18:28:59.875302 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867583    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 18:28:59.875302 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867585    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 18:28:59.875302 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867588    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 18:28:59.875302 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867591    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 18:28:59.875836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867593    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 18:28:59.875836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867595    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 18:28:59.875836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867598    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 18:28:59.875836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867600    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 18:28:59.875836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867603    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 18:28:59.875836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867606    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 18:28:59.875836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867609    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 18:28:59.875836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867611    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 18:28:59.875836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867613    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 18:28:59.875836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867616    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 18:28:59.875836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867620    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 18:28:59.875836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867623    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 18:28:59.875836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867626    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 18:28:59.875836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867629    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 18:28:59.875836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867632    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 18:28:59.875836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867635    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 18:28:59.875836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867637    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 18:28:59.875836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867640    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 18:28:59.875836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867642    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 18:28:59.876351 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867644    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 18:28:59.876351 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867647    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 18:28:59.876351 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867649    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 18:28:59.876351 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867652    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 18:28:59.876351 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867654    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 18:28:59.876351 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867656    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 18:28:59.876351 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867659    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 18:28:59.876351 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867661    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 18:28:59.876351 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867664    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 18:28:59.876351 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867668    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 18:28:59.876351 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867670    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 18:28:59.876351 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867673    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 18:28:59.876351 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867676    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 18:28:59.876351 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867678    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 18:28:59.876351 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867680    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 18:28:59.876351 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867683    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 18:28:59.876351 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867685    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 18:28:59.876351 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867688    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 18:28:59.876351 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867691    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 18:28:59.876351 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867693    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 18:28:59.876838 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867695    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 18:28:59.876838 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867698    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 18:28:59.876838 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867700    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 18:28:59.876838 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867703    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 18:28:59.876838 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867705    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 18:28:59.876838 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867708    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 18:28:59.876838 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867710    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 18:28:59.876838 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867712    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 18:28:59.876838 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867715    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 18:28:59.876838 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867718    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 18:28:59.876838 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867720    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 18:28:59.876838 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867723    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 18:28:59.876838 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867725    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 18:28:59.876838 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867727    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 18:28:59.876838 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867730    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 18:28:59.876838 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867732    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 18:28:59.876838 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867734    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 18:28:59.876838 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867737    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 18:28:59.876838 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867740    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 18:28:59.876838 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867742    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 18:28:59.877432 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867744    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 18:28:59.877432 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867748    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 18:28:59.877432 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867751    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 18:28:59.877432 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867753    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 18:28:59.877432 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867756    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 18:28:59.877432 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867758    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 18:28:59.877432 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867761    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 18:28:59.877432 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867763    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 18:28:59.877432 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867766    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 18:28:59.877432 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867769    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 18:28:59.877432 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867772    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 18:28:59.877432 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867776    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 18:28:59.877432 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867779    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 18:28:59.877432 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867781    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 18:28:59.877432 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867784    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 18:28:59.877432 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867786    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 18:28:59.877432 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867788    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 18:28:59.877432 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867791    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 18:28:59.877432 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867794    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 18:28:59.877432 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867797    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 18:28:59.877911 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867801    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 18:28:59.877911 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867804    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 18:28:59.877911 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867807    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 18:28:59.877911 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867809    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 18:28:59.877911 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867812    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 18:28:59.877911 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867815    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 18:28:59.877911 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.867817    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 18:28:59.877911 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.868854    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 18:28:59.877911 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.875895    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 18:28:59.877911 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.875910    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 18:28:59.877911 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.875930    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 18:28:59.877911 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.875936    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 18:28:59.877911 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.875992    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 18:28:59.877911 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.875997    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 18:28:59.877911 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876000    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 18:28:59.878301 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876004    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 18:28:59.878301 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876007    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 18:28:59.878301 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876009    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 18:28:59.878301 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876012    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 18:28:59.878301 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876015    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 18:28:59.878301 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876017    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 18:28:59.878301 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876019    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 18:28:59.878301 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876022    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 18:28:59.878301 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876024    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 18:28:59.878301 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876027    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 18:28:59.878301 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876029    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 18:28:59.878301 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876032    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 18:28:59.878301 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876035    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 18:28:59.878301 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876037    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 18:28:59.878301 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876040    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 18:28:59.878301 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876042    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 18:28:59.878301 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876045    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 18:28:59.878301 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876048    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 18:28:59.878301 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876050    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 18:28:59.878301 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876052    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 18:28:59.878791 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876055    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 18:28:59.878791 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876058    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 18:28:59.878791 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876060    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 18:28:59.878791 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876063    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 18:28:59.878791 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876066    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 18:28:59.878791 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876068    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 18:28:59.878791 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876072    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 18:28:59.878791 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876075    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 18:28:59.878791 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876077    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 18:28:59.878791 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876079    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 18:28:59.878791 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876082    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 18:28:59.878791 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876084    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 18:28:59.878791 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876087    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 18:28:59.878791 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876089    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 18:28:59.878791 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876091    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 18:28:59.878791 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876094    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 18:28:59.878791 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876096    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 18:28:59.878791 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876099    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 18:28:59.878791 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876101    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 18:28:59.878791 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876104    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 18:28:59.879347 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876108    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 18:28:59.879347 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876112    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 18:28:59.879347 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876138    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 18:28:59.879347 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876143    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 18:28:59.879347 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876146    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 18:28:59.879347 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876149    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 18:28:59.879347 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876152    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 18:28:59.879347 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876154    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 18:28:59.879347 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876157    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 18:28:59.879347 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876160    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 18:28:59.879347 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876163    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 18:28:59.879347 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876166    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 18:28:59.879347 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876169    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 18:28:59.879347 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876172    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 18:28:59.879347 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876175    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 18:28:59.879347 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876179    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 18:28:59.879347 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876182    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 18:28:59.879347 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876185    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 18:28:59.879347 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876188    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 18:28:59.879836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876192    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 18:28:59.879836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876194    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 18:28:59.879836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876197    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 18:28:59.879836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876200    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 18:28:59.879836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876202    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 18:28:59.879836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876205    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 18:28:59.879836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876207    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 18:28:59.879836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876209    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 18:28:59.879836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876211    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 18:28:59.879836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876214    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 18:28:59.879836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876217    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 18:28:59.879836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876219    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 18:28:59.879836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876222    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 18:28:59.879836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876224    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 18:28:59.879836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876226    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 18:28:59.879836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876229    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 18:28:59.879836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876231    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 18:28:59.879836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876233    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 18:28:59.879836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876236    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 18:28:59.879836 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876238    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 18:28:59.880336 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876241    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 18:28:59.880336 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876243    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 18:28:59.880336 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876246    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 18:28:59.880336 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876249    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 18:28:59.880336 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876252    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 18:28:59.880336 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876255    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 18:28:59.880336 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876257    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 18:28:59.880336 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876260    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 18:28:59.880336 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876262    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 18:28:59.880336 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876265    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 18:28:59.880336 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876267    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 18:28:59.880336 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876270    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 18:28:59.880336 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876272    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 18:28:59.880336 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876275    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 18:28:59.880336 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876277    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 18:28:59.880336 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876280    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 18:28:59.880336 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876283    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 18:28:59.880773 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.876288    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 18:28:59.880773 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876406    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 18:28:59.880773 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876410    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 18:28:59.880773 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876413    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 18:28:59.880773 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876416    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 18:28:59.880773 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876419    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 18:28:59.880773 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876421    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 18:28:59.880773 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876424    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 18:28:59.880773 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876426    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 18:28:59.880773 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876429    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 18:28:59.880773 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876431    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 18:28:59.880773 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876434    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 18:28:59.880773 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876437    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 18:28:59.880773 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876439    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 18:28:59.880773 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876442    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 18:28:59.881157 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876444    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 18:28:59.881157 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876446    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 18:28:59.881157 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876449    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 18:28:59.881157 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876451    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 18:28:59.881157 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876454    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 18:28:59.881157 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876456    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 18:28:59.881157 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876459    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 18:28:59.881157 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876462    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 18:28:59.881157 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876464    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 18:28:59.881157 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876467    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 18:28:59.881157 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876470    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 18:28:59.881157 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876474    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 18:28:59.881157 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876477    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 18:28:59.881157 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876479    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 18:28:59.881157 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876483    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 18:28:59.881157 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876485    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 18:28:59.881157 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876488    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 18:28:59.881157 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876490    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 18:28:59.881157 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876492    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 18:28:59.881631 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876495    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 18:28:59.881631 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876497    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 18:28:59.881631 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876500    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 18:28:59.881631 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876502    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 18:28:59.881631 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876505    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 18:28:59.881631 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876507    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 18:28:59.881631 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876510    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 18:28:59.881631 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876512    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 18:28:59.881631 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876515    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 18:28:59.881631 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876517    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 18:28:59.881631 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876519    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 18:28:59.881631 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876522    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 18:28:59.881631 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876525    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 18:28:59.881631 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876527    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 18:28:59.881631 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876530    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 18:28:59.881631 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876532    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 18:28:59.881631 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876534    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 18:28:59.881631 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876537    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 18:28:59.881631 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876539    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 18:28:59.881631 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876542    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 18:28:59.882112 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876544    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 18:28:59.882112 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876546    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 18:28:59.882112 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876549    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 18:28:59.882112 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876551    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 18:28:59.882112 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876553    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 18:28:59.882112 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876556    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 18:28:59.882112 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876559    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 18:28:59.882112 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876561    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 18:28:59.882112 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876564    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 18:28:59.882112 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876566    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 18:28:59.882112 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876569    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 18:28:59.882112 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876571    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 18:28:59.882112 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876574    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 18:28:59.882112 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876576    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 18:28:59.882112 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876578    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 18:28:59.882112 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876581    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 18:28:59.882112 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876583    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 18:28:59.882112 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876586    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 18:28:59.882112 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876588    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 18:28:59.882112 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876591    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 18:28:59.882622 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876593    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 18:28:59.882622 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876596    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 18:28:59.882622 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876598    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 18:28:59.882622 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876601    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 18:28:59.882622 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876603    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 18:28:59.882622 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876606    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 18:28:59.882622 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876608    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 18:28:59.882622 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876610    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 18:28:59.882622 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876613    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 18:28:59.882622 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876615    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 18:28:59.882622 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876618    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 18:28:59.882622 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876620    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 18:28:59.882622 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876623    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 18:28:59.882622 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876626    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 18:28:59.882622 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876630    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 18:28:59.882622 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876632    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 18:28:59.882622 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876635    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 18:28:59.882622 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876637    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 18:28:59.882622 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876640    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 18:28:59.882622 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876642    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 18:28:59.883157 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876644    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 18:28:59.883157 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876648    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 18:28:59.883157 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876652    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 18:28:59.883157 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876654    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 18:28:59.883157 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876657    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 18:28:59.883157 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:28:59.876660    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 18:28:59.883157 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.876664    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 18:28:59.883157 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.876773    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 18:28:59.883157 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.878878    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 18:28:59.883157 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.880613    2566 server.go:1019] "Starting client certificate rotation"
Jul 27 18:28:59.883157 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.880712    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 18:28:59.883157 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.880747    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 18:28:59.883157 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.883049    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 18:28:59.886624 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.886606    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 18:28:59.899481 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.899454    2566 log.go:25] "Validated CRI v1 runtime API"
Jul 27 18:28:59.901673 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.901655    2566 log.go:25] "Validated CRI v1 image API"
Jul 27 18:28:59.902425 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.902408    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 18:28:59.905269 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.905249    2566 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8294c853-68c6-47a2-a555-a05d63d9f859:/dev/nvme0n1p3 d911461e-08b3-435b-9781-c9d82b1c69e8:/dev/nvme0n1p4]
Jul 27 18:28:59.905316 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.905269    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 18:28:59.911412 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.911292    2566 manager.go:216] Machine: {Timestamp:2026-07-27 18:28:59.909843037 +0000 UTC m=+0.344100639 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099501 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2639656973233d1bbc92276290ef0f SystemUUID:ec263965-6973-233d-1bbc-92276290ef0f BootID:43ae4270-a0e1-4349-bf97-965e0bf6f739 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1a:b2:81:29:39 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1a:b2:81:29:39 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:92:be:f1:77:85 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 18:28:59.911412 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.911400    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 18:28:59.911533 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.911478    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 18:28:59.913101 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.913076    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 18:28:59.914612 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.914584    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 18:28:59.914763 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.914617    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-196.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 18:28:59.916529 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.916518    2566 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 18:28:59.916566 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.916534    2566 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 18:28:59.916566 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.916542    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 18:28:59.917415 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.917405    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 18:28:59.918840 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.918830    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 18:28:59.922112 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.922100    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 18:28:59.922245 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.922235    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 18:28:59.922277 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.922267    2566 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 18:28:59.922277 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.922276    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 18:28:59.922334 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.922291    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 18:28:59.922334 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.922300    2566 kubelet.go:398] "Adding apiserver pod source"
Jul 27 18:28:59.922334 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.922308    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 18:28:59.922666 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.922655    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 18:28:59.922701 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.922673    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 18:28:59.925923 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.925904    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 18:28:59.927368 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.927353    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 18:28:59.927440 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.927384    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 18:28:59.928209 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.928198    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 18:28:59.928242 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.928216    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 18:28:59.928242 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.928222    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 18:28:59.928242 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.928229    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 18:28:59.928324 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.928243    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 18:28:59.928324 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.928251    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 18:28:59.928324 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.928259    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 18:28:59.928324 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.928264    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 18:28:59.928324 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.928270    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 18:28:59.928324 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.928275    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 18:28:59.928324 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.928289    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 18:28:59.928324 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.928295    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 18:28:59.929288 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.929271    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 18:28:59.929288 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.929288    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 18:28:59.933069 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.933054    2566 server.go:1268] "Started kubelet"
Jul 27 18:28:59.933167 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.933152    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-196.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 18:28:59.933235 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.933157    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 18:28:59.933235 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.933212    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 18:28:59.933327 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.933257    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 18:28:59.933839 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:28:59.933807    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-196.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 18:28:59.933839 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:28:59.933825    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 18:28:59.934033 ip-10-0-128-196 systemd[1]: Started Kubernetes Kubelet.
Jul 27 18:28:59.934813 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.934797    2566 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 18:28:59.935090 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.934905    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 18:28:59.939995 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:28:59.939745    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 18:28:59.943041 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:28:59.941758    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-196.ec2.internal.18c638f7f5ed49c4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-196.ec2.internal,UID:ip-10-0-128-196.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-196.ec2.internal,},FirstTimestamp:2026-07-27 18:28:59.933026756 +0000 UTC m=+0.367284358,LastTimestamp:2026-07-27 18:28:59.933026756 +0000 UTC m=+0.367284358,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-196.ec2.internal,}"
Jul 27 18:28:59.945682 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.945665    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 18:28:59.945761 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.945720    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 18:28:59.945814 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:28:59.945790    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-196.ec2.internal\" not found"
Jul 27 18:28:59.946448 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.946431    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 18:28:59.946448 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.946443    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 18:28:59.946599 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.946458    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 18:28:59.946599 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.946590    2566 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 18:28:59.946599 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.946598    2566 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 18:28:59.948260 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.948243    2566 factory.go:153] Registering CRI-O factory
Jul 27 18:28:59.948260 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.948261    2566 factory.go:223] Registration of the crio container factory successfully
Jul 27 18:28:59.948383 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.948312    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 18:28:59.948383 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.948322    2566 factory.go:55] Registering systemd factory
Jul 27 18:28:59.948383 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.948331    2566 factory.go:223] Registration of the systemd container factory successfully
Jul 27 18:28:59.948383 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.948352    2566 factory.go:103] Registering Raw factory
Jul 27 18:28:59.948383 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.948367    2566 manager.go:1196] Started watching for new ooms in manager
Jul 27 18:28:59.948944 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.948926    2566 manager.go:349] Starting recovery of all containers
Jul 27 18:28:59.951950 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.951925    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t6hfd"
Jul 27 18:28:59.957409 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:28:59.957363    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 27 18:28:59.957409 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:28:59.957386    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-196.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 27 18:28:59.960627 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.960609    2566 manager.go:354] Recovery completed
Jul 27 18:28:59.961465 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.961338    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t6hfd"
Jul 27 18:28:59.965939 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.965925    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 18:28:59.968422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.968403    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-196.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 18:28:59.968489 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.968437    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-196.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 18:28:59.968489 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.968452    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-196.ec2.internal" event="NodeHasSufficientPID"
Jul 27 18:28:59.969029 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.969007    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 18:28:59.969029 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.969022    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 18:28:59.969029 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.969014    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 18:28:59.969219 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.969039    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 18:28:59.969818 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.969796    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-196.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 18:28:59.969887 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.969829    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-196.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 18:28:59.969887 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.969840    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-196.ec2.internal" event="NodeHasSufficientPID"
Jul 27 18:28:59.970284 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.970270    2566 policy_none.go:49] "None policy: Start"
Jul 27 18:28:59.970332 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.970293    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 18:28:59.970332 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.970310    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 18:28:59.971575 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:28:59.971563    2566 policy_none.go:47] "Start"
Jul 27 18:29:00.006770 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.006752    2566 manager.go:337] "Starting Device Plugin manager"
Jul 27 18:29:00.006873 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:00.006795    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 18:29:00.006873 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.006810    2566 server.go:87] "Starting device plugin registration server"
Jul 27 18:29:00.008300 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.007885    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 18:29:00.008300 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.007915    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 18:29:00.008300 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.008256    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 18:29:00.008513 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.008391    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 18:29:00.008513 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.008402    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 18:29:00.010237 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:00.010203    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 18:29:00.010562 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:00.010544    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-196.ec2.internal\" not found"
Jul 27 18:29:00.108380 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.108341    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 18:29:00.109661 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.109535    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-196.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 18:29:00.109661 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.109564    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-196.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 18:29:00.109661 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.109577    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-196.ec2.internal" event="NodeHasSufficientPID"
Jul 27 18:29:00.109661 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.109614    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-196.ec2.internal"
Jul 27 18:29:00.115812 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.115784    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 18:29:00.117098 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.117078    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 18:29:00.117180 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.117151    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 18:29:00.117240 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.117180    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 18:29:00.117288 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:00.117236    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 18:29:00.118262 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.118245    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-196.ec2.internal"
Jul 27 18:29:00.118332 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:00.118270    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-196.ec2.internal\": node \"ip-10-0-128-196.ec2.internal\" not found"
Jul 27 18:29:00.119313 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.119297    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 18:29:00.171302 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:00.171268    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-196.ec2.internal\" not found"
Jul 27 18:29:00.217394 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.217346    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-196.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-196.ec2.internal"]
Jul 27 18:29:00.217562 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.217490    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 18:29:00.218482 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.218464    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-196.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 18:29:00.218549 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.218495    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-196.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 18:29:00.218549 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.218505    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-196.ec2.internal" event="NodeHasSufficientPID"
Jul 27 18:29:00.220185 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.220173    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 18:29:00.220284 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.220269    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-196.ec2.internal"
Jul 27 18:29:00.220346 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.220297    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 18:29:00.220915 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.220897    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-196.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 18:29:00.220987 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.220930    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-196.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 18:29:00.220987 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.220941    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-196.ec2.internal" event="NodeHasSufficientPID"
Jul 27 18:29:00.220987 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.220897    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-196.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 18:29:00.221091 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.221011    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-196.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 18:29:00.221091 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.221026    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-196.ec2.internal" event="NodeHasSufficientPID"
Jul 27 18:29:00.222321 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.222303    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-196.ec2.internal"
Jul 27 18:29:00.222408 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.222329    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 18:29:00.222963 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.222946    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-196.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 18:29:00.223018 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.222974    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-196.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 18:29:00.223018 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.222990    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-196.ec2.internal" event="NodeHasSufficientPID"
Jul 27 18:29:00.245901 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:00.245880    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-196.ec2.internal\" not found" node="ip-10-0-128-196.ec2.internal"
Jul 27 18:29:00.247966 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.247948    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b20c1cf26d726b4ecc2d495ed14b7a50-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-196.ec2.internal\" (UID: \"b20c1cf26d726b4ecc2d495ed14b7a50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-196.ec2.internal"
Jul 27 18:29:00.248065 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.247971    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b20c1cf26d726b4ecc2d495ed14b7a50-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-196.ec2.internal\" (UID: \"b20c1cf26d726b4ecc2d495ed14b7a50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-196.ec2.internal"
Jul 27 18:29:00.248065 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.247991    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7cd8d2439d53474a3dfd84498491b9ab-config\") pod \"kube-apiserver-proxy-ip-10-0-128-196.ec2.internal\" (UID: \"7cd8d2439d53474a3dfd84498491b9ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-196.ec2.internal"
Jul 27 18:29:00.250467 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:00.250451    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-196.ec2.internal\" not found" node="ip-10-0-128-196.ec2.internal"
Jul 27 18:29:00.271908 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:00.271887    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-196.ec2.internal\" not found"
Jul 27 18:29:00.348655 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.348627    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b20c1cf26d726b4ecc2d495ed14b7a50-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-196.ec2.internal\" (UID: \"b20c1cf26d726b4ecc2d495ed14b7a50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-196.ec2.internal"
Jul 27 18:29:00.348752 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.348659    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7cd8d2439d53474a3dfd84498491b9ab-config\") pod \"kube-apiserver-proxy-ip-10-0-128-196.ec2.internal\" (UID: \"7cd8d2439d53474a3dfd84498491b9ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-196.ec2.internal"
Jul 27 18:29:00.348752 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.348677    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b20c1cf26d726b4ecc2d495ed14b7a50-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-196.ec2.internal\" (UID: \"b20c1cf26d726b4ecc2d495ed14b7a50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-196.ec2.internal"
Jul 27 18:29:00.348752 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.348720    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b20c1cf26d726b4ecc2d495ed14b7a50-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-196.ec2.internal\" (UID: \"b20c1cf26d726b4ecc2d495ed14b7a50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-196.ec2.internal"
Jul 27 18:29:00.348752 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.348731    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b20c1cf26d726b4ecc2d495ed14b7a50-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-196.ec2.internal\" (UID: \"b20c1cf26d726b4ecc2d495ed14b7a50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-196.ec2.internal"
Jul 27 18:29:00.348752 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.348737    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7cd8d2439d53474a3dfd84498491b9ab-config\") pod \"kube-apiserver-proxy-ip-10-0-128-196.ec2.internal\" (UID: \"7cd8d2439d53474a3dfd84498491b9ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-196.ec2.internal"
Jul 27 18:29:00.373050 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:00.372988    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-196.ec2.internal\" not found"
Jul 27 18:29:00.473828 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:00.473796    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-196.ec2.internal\" not found"
Jul 27 18:29:00.548203 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.548166    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-196.ec2.internal"
Jul 27 18:29:00.553997 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.553978    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-196.ec2.internal"
Jul 27 18:29:00.574718 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:00.574691    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-196.ec2.internal\" not found"
Jul 27 18:29:00.675300 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:00.675210    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-196.ec2.internal\" not found"
Jul 27 18:29:00.775854 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:00.775829    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-196.ec2.internal\" not found"
Jul 27 18:29:00.805891 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.805863    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 18:29:00.877158 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:00.877101    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-196.ec2.internal\" not found"
Jul 27 18:29:00.879385 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.879363    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 18:29:00.879552 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.879529    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 18:29:00.879601 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.879548    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 18:29:00.898433 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.898405    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 18:29:00.912097 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.912076    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 18:29:00.923466 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.923447    2566 apiserver.go:52] "Watching apiserver"
Jul 27 18:29:00.933970 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.933910    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 18:29:00.934327 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.934306    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv","openshift-dns/node-resolver-cl7cw","openshift-image-registry/node-ca-4g7qz","openshift-multus/multus-additional-cni-plugins-qjwvr","openshift-multus/multus-z6tg2","kube-system/konnectivity-agent-l6rrw","openshift-cluster-node-tuning-operator/tuned-zqkqx","openshift-multus/network-metrics-daemon-qc4qw","openshift-network-diagnostics/network-check-target-jpq2j","openshift-network-operator/iptables-alerter-f8mj4","openshift-ovn-kubernetes/ovnkube-node-tdb9t","kube-system/global-pull-secret-syncer-qrfwk"]
Jul 27 18:29:00.936097 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.936076    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv"
Jul 27 18:29:00.937468 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.937454    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cl7cw"
Jul 27 18:29:00.939121 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.939096    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4g7qz"
Jul 27 18:29:00.939266 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.939248    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 18:29:00.939335 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.939309    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 18:29:00.939423 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.939355    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 18:29:00.939423 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.939395    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cqfq8\""
Jul 27 18:29:00.940234 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.940218    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 18:29:00.940430 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.940416    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 18:29:00.940536 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.940425    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cc5kb\""
Jul 27 18:29:00.943681 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.941038    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qjwvr"
Jul 27 18:29:00.943681 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.942705    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 18:29:00.943681 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.942762    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-msf2r\""
Jul 27 18:29:00.943681 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.942949    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 18:29:00.943681 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.943017    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 18:29:00.943974 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.943959    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mcct4\""
Jul 27 18:29:00.944060 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.944044    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:00.944510 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.944446    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 18:29:00.944793 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.944445    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 18:29:00.944866 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.944846    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 18:29:00.944934 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.944886    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 18:29:00.944934 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.944915    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 18:29:00.945806 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.945790    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-196.ec2.internal"
Jul 27 18:29:00.945895 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.945876    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 18:29:00.945977 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.945881    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l6rrw"
Jul 27 18:29:00.947507 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.947485    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:00.947600 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.947551    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 18:29:00.947662 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.947556    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vmnbn\""
Jul 27 18:29:00.948924 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.948909    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dxz2n\""
Jul 27 18:29:00.948985 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.948938    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 18:29:00.948985 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.948970    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 18:29:00.949069 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.949001    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:00.949106 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:00.949087    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:00.950045 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.950029    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-stdww\""
Jul 27 18:29:00.950111 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.950068    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 18:29:00.950209 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.950190    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 18:29:00.950486 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.950472    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:00.950540 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:00.950524    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:00.952058 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952042    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f8mj4"
Jul 27 18:29:00.952220 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952187    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-cnibin\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:00.952288 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952239    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-multus-socket-dir-parent\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:00.952288 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952266    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dff73097-b755-4894-bbd4-8b3a3a69d3a8-tmp\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:00.952288 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952285    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b81bd2e0-40bf-41ba-b0ab-2c86fdee9438-socket-dir\") pod \"aws-ebs-csi-driver-node-bxjrv\" (UID: \"b81bd2e0-40bf-41ba-b0ab-2c86fdee9438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv"
Jul 27 18:29:00.952414 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952326    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-var-lib-kubelet\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:00.952414 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952384    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-hostroot\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:00.952497 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952425    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18a80374-61a1-4cd0-806d-9578a7e70433-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qjwvr\" (UID: \"18a80374-61a1-4cd0-806d-9578a7e70433\") " pod="openshift-multus/multus-additional-cni-plugins-qjwvr"
Jul 27 18:29:00.952497 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952460    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7prjs\" (UniqueName: \"kubernetes.io/projected/dff73097-b755-4894-bbd4-8b3a3a69d3a8-kube-api-access-7prjs\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:00.952497 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952477    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/18a80374-61a1-4cd0-806d-9578a7e70433-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qjwvr\" (UID: \"18a80374-61a1-4cd0-806d-9578a7e70433\") " pod="openshift-multus/multus-additional-cni-plugins-qjwvr"
Jul 27 18:29:00.952497 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952496    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-host-var-lib-cni-bin\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:00.952659 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952511    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-etc-systemd\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:00.952659 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952537    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-host\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:00.952659 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952553    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b81bd2e0-40bf-41ba-b0ab-2c86fdee9438-etc-selinux\") pod \"aws-ebs-csi-driver-node-bxjrv\" (UID: \"b81bd2e0-40bf-41ba-b0ab-2c86fdee9438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv"
Jul 27 18:29:00.952659 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952569    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-cni-binary-copy\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:00.952659 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952584    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-host-run-k8s-cni-cncf-io\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:00.952659 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952602    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-sys\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:00.952659 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952616    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-lib-modules\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:00.952659 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952630    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b81bd2e0-40bf-41ba-b0ab-2c86fdee9438-sys-fs\") pod \"aws-ebs-csi-driver-node-bxjrv\" (UID: \"b81bd2e0-40bf-41ba-b0ab-2c86fdee9438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv"
Jul 27 18:29:00.952659 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952651    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-host-run-netns\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:00.952965 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952667    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5b7a8e0f-9255-415f-b6fe-123c07105bc3-konnectivity-ca\") pod \"konnectivity-agent-l6rrw\" (UID: \"5b7a8e0f-9255-415f-b6fe-123c07105bc3\") " pod="kube-system/konnectivity-agent-l6rrw"
Jul 27 18:29:00.952965 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952695    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmxlh\" (UniqueName: \"kubernetes.io/projected/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-kube-api-access-tmxlh\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:00.952965 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952716    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dff73097-b755-4894-bbd4-8b3a3a69d3a8-etc-tuned\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:00.952965 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952732    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4378195f-ed99-42aa-90e6-0fde9f5394f1-hosts-file\") pod \"node-resolver-cl7cw\" (UID: \"4378195f-ed99-42aa-90e6-0fde9f5394f1\") " pod="openshift-dns/node-resolver-cl7cw"
Jul 27 18:29:00.952965 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952746    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-multus-daemon-config\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:00.952965 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952760    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-etc-sysctl-d\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:00.952965 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952788    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6wds\" (UniqueName: \"kubernetes.io/projected/4378195f-ed99-42aa-90e6-0fde9f5394f1-kube-api-access-r6wds\") pod \"node-resolver-cl7cw\" (UID: \"4378195f-ed99-42aa-90e6-0fde9f5394f1\") " pod="openshift-dns/node-resolver-cl7cw"
Jul 27 18:29:00.952965 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952821    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18a80374-61a1-4cd0-806d-9578a7e70433-cnibin\") pod \"multus-additional-cni-plugins-qjwvr\" (UID: \"18a80374-61a1-4cd0-806d-9578a7e70433\") " pod="openshift-multus/multus-additional-cni-plugins-qjwvr"
Jul 27 18:29:00.952965 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952839    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4h2n\" (UniqueName: \"kubernetes.io/projected/0806ed23-3336-4327-ae3a-0da1e826f856-kube-api-access-w4h2n\") pod \"node-ca-4g7qz\" (UID: \"0806ed23-3336-4327-ae3a-0da1e826f856\") " pod="openshift-image-registry/node-ca-4g7qz"
Jul 27 18:29:00.952965 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952868    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdgzn\" (UniqueName: \"kubernetes.io/projected/b81bd2e0-40bf-41ba-b0ab-2c86fdee9438-kube-api-access-vdgzn\") pod \"aws-ebs-csi-driver-node-bxjrv\" (UID: \"b81bd2e0-40bf-41ba-b0ab-2c86fdee9438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv"
Jul 27 18:29:00.952965 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952886    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-multus-cni-dir\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:00.952965 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952900    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-host-run-multus-certs\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:00.952965 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952916    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18a80374-61a1-4cd0-806d-9578a7e70433-system-cni-dir\") pod \"multus-additional-cni-plugins-qjwvr\" (UID: \"18a80374-61a1-4cd0-806d-9578a7e70433\") " pod="openshift-multus/multus-additional-cni-plugins-qjwvr"
Jul 27 18:29:00.952965 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952938    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-645kg\" (UniqueName: \"kubernetes.io/projected/18a80374-61a1-4cd0-806d-9578a7e70433-kube-api-access-645kg\") pod \"multus-additional-cni-plugins-qjwvr\" (UID: \"18a80374-61a1-4cd0-806d-9578a7e70433\") " pod="openshift-multus/multus-additional-cni-plugins-qjwvr"
Jul 27 18:29:00.952965 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952958    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0806ed23-3336-4327-ae3a-0da1e826f856-host\") pod \"node-ca-4g7qz\" (UID: \"0806ed23-3336-4327-ae3a-0da1e826f856\") " pod="openshift-image-registry/node-ca-4g7qz"
Jul 27 18:29:00.953439 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952973    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0806ed23-3336-4327-ae3a-0da1e826f856-serviceca\") pod \"node-ca-4g7qz\" (UID: \"0806ed23-3336-4327-ae3a-0da1e826f856\") " pod="openshift-image-registry/node-ca-4g7qz"
Jul 27 18:29:00.953439 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.952994    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-host-var-lib-cni-multus\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:00.953439 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.953024    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-multus-conf-dir\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:00.953439 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.953051    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-etc-sysconfig\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:00.953439 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.953071    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/18a80374-61a1-4cd0-806d-9578a7e70433-cni-binary-copy\") pod \"multus-additional-cni-plugins-qjwvr\" (UID: \"18a80374-61a1-4cd0-806d-9578a7e70433\") " pod="openshift-multus/multus-additional-cni-plugins-qjwvr"
Jul 27 18:29:00.953439 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.953112    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-system-cni-dir\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:00.953439 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.953141    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5b7a8e0f-9255-415f-b6fe-123c07105bc3-agent-certs\") pod \"konnectivity-agent-l6rrw\" (UID: \"5b7a8e0f-9255-415f-b6fe-123c07105bc3\") " pod="kube-system/konnectivity-agent-l6rrw"
Jul 27 18:29:00.953439 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.953166    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-etc-kubernetes\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:00.953439 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.953183    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-os-release\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:00.953439 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.953199    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-etc-sysctl-conf\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:00.953439 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.953229    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b81bd2e0-40bf-41ba-b0ab-2c86fdee9438-registration-dir\") pod \"aws-ebs-csi-driver-node-bxjrv\" (UID: \"b81bd2e0-40bf-41ba-b0ab-2c86fdee9438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv"
Jul 27 18:29:00.953439 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.953249    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b81bd2e0-40bf-41ba-b0ab-2c86fdee9438-device-dir\") pod \"aws-ebs-csi-driver-node-bxjrv\" (UID: \"b81bd2e0-40bf-41ba-b0ab-2c86fdee9438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv"
Jul 27 18:29:00.953439 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.953272    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4378195f-ed99-42aa-90e6-0fde9f5394f1-tmp-dir\") pod \"node-resolver-cl7cw\" (UID: \"4378195f-ed99-42aa-90e6-0fde9f5394f1\") " pod="openshift-dns/node-resolver-cl7cw"
Jul 27 18:29:00.953439 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.953289    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/18a80374-61a1-4cd0-806d-9578a7e70433-os-release\") pod \"multus-additional-cni-plugins-qjwvr\" (UID: \"18a80374-61a1-4cd0-806d-9578a7e70433\") " pod="openshift-multus/multus-additional-cni-plugins-qjwvr"
Jul 27 18:29:00.953439 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.953317    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/18a80374-61a1-4cd0-806d-9578a7e70433-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qjwvr\" (UID: \"18a80374-61a1-4cd0-806d-9578a7e70433\") " pod="openshift-multus/multus-additional-cni-plugins-qjwvr"
Jul 27 18:29:00.953439 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.953343    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-host-var-lib-kubelet\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:00.954059 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.953358    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-etc-kubernetes\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:00.954059 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.953398    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-etc-modprobe-d\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:00.954059 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.953421    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-run\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:00.954059 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.953447    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b81bd2e0-40bf-41ba-b0ab-2c86fdee9438-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bxjrv\" (UID: \"b81bd2e0-40bf-41ba-b0ab-2c86fdee9438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv"
Jul 27 18:29:00.954059 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.953932    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:00.954622 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.954602    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vsnt9\""
Jul 27 18:29:00.954665 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.954642    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 18:29:00.954791 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.954777    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 18:29:00.954828 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.954776    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 18:29:00.955747 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.955729    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:00.955827 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:00.955793    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:00.956688 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.956673    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 18:29:00.957291 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.957275    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7gqrv\""
Jul 27 18:29:00.957670 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.957652    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 18:29:00.957817 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.957800    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-196.ec2.internal"]
Jul 27 18:29:00.958462 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.958149    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 18:29:00.958462 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.958165    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 18:29:00.958462 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.958182    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 18:29:00.958462 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.958234    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 18:29:00.958462 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.958245    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 18:29:00.958462 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.958248    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-196.ec2.internal"
Jul 27 18:29:00.963010 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.962992    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 18:29:00.966508 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.966448    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 18:24:00 +0000 UTC" deadline="2028-05-10 00:42:34.644140906 +0000 UTC"
Jul 27 18:29:00.966588 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.966509    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15654h13m33.677636162s"
Jul 27 18:29:00.968060 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.968038    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 18:29:00.968252 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.968236    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-196.ec2.internal"]
Jul 27 18:29:00.983705 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.983683    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9kncf"
Jul 27 18:29:00.992364 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:00.992347    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9kncf"
Jul 27 18:29:01.047330 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.047309    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 18:29:01.053968 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.053949    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-host-run-k8s-cni-cncf-io\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.054021 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.053979    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-sys\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.054021 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054000    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-lib-modules\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.054021 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054017    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b81bd2e0-40bf-41ba-b0ab-2c86fdee9438-sys-fs\") pod \"aws-ebs-csi-driver-node-bxjrv\" (UID: \"b81bd2e0-40bf-41ba-b0ab-2c86fdee9438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv"
Jul 27 18:29:01.054170 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054036    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-host-run-netns\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.054170 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054062    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-host-run-netns\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.054170 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054059    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-host-run-k8s-cni-cncf-io\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.054170 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054088    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5b7a8e0f-9255-415f-b6fe-123c07105bc3-konnectivity-ca\") pod \"konnectivity-agent-l6rrw\" (UID: \"5b7a8e0f-9255-415f-b6fe-123c07105bc3\") " pod="kube-system/konnectivity-agent-l6rrw"
Jul 27 18:29:01.054170 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054115    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcsl5\" (UniqueName: \"kubernetes.io/projected/8c44c98c-c8de-4a6e-b827-3d56a456f1eb-kube-api-access-pcsl5\") pod \"iptables-alerter-f8mj4\" (UID: \"8c44c98c-c8de-4a6e-b827-3d56a456f1eb\") " pod="openshift-network-operator/iptables-alerter-f8mj4"
Jul 27 18:29:01.054170 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054136    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-host-run-netns\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.054170 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054083    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-sys\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.054170 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054147    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b81bd2e0-40bf-41ba-b0ab-2c86fdee9438-sys-fs\") pod \"aws-ebs-csi-driver-node-bxjrv\" (UID: \"b81bd2e0-40bf-41ba-b0ab-2c86fdee9438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv"
Jul 27 18:29:01.054170 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054159    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fa213400-5979-496e-9d82-f6ba6a0f0333-ovnkube-script-lib\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.054706 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054173    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-lib-modules\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.054706 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054202    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmxlh\" (UniqueName: \"kubernetes.io/projected/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-kube-api-access-tmxlh\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.054706 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054229    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dff73097-b755-4894-bbd4-8b3a3a69d3a8-etc-tuned\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.054706 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054248    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4378195f-ed99-42aa-90e6-0fde9f5394f1-hosts-file\") pod \"node-resolver-cl7cw\" (UID: \"4378195f-ed99-42aa-90e6-0fde9f5394f1\") " pod="openshift-dns/node-resolver-cl7cw"
Jul 27 18:29:01.054706 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054264    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-multus-daemon-config\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.054706 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054287    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-etc-sysctl-d\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.054706 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054304    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6wds\" (UniqueName: \"kubernetes.io/projected/4378195f-ed99-42aa-90e6-0fde9f5394f1-kube-api-access-r6wds\") pod \"node-resolver-cl7cw\" (UID: \"4378195f-ed99-42aa-90e6-0fde9f5394f1\") " pod="openshift-dns/node-resolver-cl7cw"
Jul 27 18:29:01.054706 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054306    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4378195f-ed99-42aa-90e6-0fde9f5394f1-hosts-file\") pod \"node-resolver-cl7cw\" (UID: \"4378195f-ed99-42aa-90e6-0fde9f5394f1\") " pod="openshift-dns/node-resolver-cl7cw"
Jul 27 18:29:01.054706 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054337    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18a80374-61a1-4cd0-806d-9578a7e70433-cnibin\") pod \"multus-additional-cni-plugins-qjwvr\" (UID: \"18a80374-61a1-4cd0-806d-9578a7e70433\") " pod="openshift-multus/multus-additional-cni-plugins-qjwvr"
Jul 27 18:29:01.054706 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054367    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4h2n\" (UniqueName: \"kubernetes.io/projected/0806ed23-3336-4327-ae3a-0da1e826f856-kube-api-access-w4h2n\") pod \"node-ca-4g7qz\" (UID: \"0806ed23-3336-4327-ae3a-0da1e826f856\") " pod="openshift-image-registry/node-ca-4g7qz"
Jul 27 18:29:01.054706 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054397    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdgzn\" (UniqueName: \"kubernetes.io/projected/b81bd2e0-40bf-41ba-b0ab-2c86fdee9438-kube-api-access-vdgzn\") pod \"aws-ebs-csi-driver-node-bxjrv\" (UID: \"b81bd2e0-40bf-41ba-b0ab-2c86fdee9438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv"
Jul 27 18:29:01.054706 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054426    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vnls\" (UniqueName: \"kubernetes.io/projected/6751c8be-8944-44d2-8e25-f3fd6926225e-kube-api-access-2vnls\") pod \"network-metrics-daemon-qc4qw\" (UID: \"6751c8be-8944-44d2-8e25-f3fd6926225e\") " pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:01.054706 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054455    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dftwx\" (UniqueName: \"kubernetes.io/projected/53874b39-22ec-4554-ab95-643a49ee8699-kube-api-access-dftwx\") pod \"network-check-target-jpq2j\" (UID: \"53874b39-22ec-4554-ab95-643a49ee8699\") " pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:01.054706 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054480    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-systemd-units\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.054706 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054490    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-etc-sysctl-d\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.054706 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054503    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-host-slash\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.054706 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054537    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-run-systemd\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.055555 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054569    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-multus-cni-dir\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.055555 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054599    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-host-run-multus-certs\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.055555 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054609    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18a80374-61a1-4cd0-806d-9578a7e70433-cnibin\") pod \"multus-additional-cni-plugins-qjwvr\" (UID: \"18a80374-61a1-4cd0-806d-9578a7e70433\") " pod="openshift-multus/multus-additional-cni-plugins-qjwvr"
Jul 27 18:29:01.055555 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054625    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18a80374-61a1-4cd0-806d-9578a7e70433-system-cni-dir\") pod \"multus-additional-cni-plugins-qjwvr\" (UID: \"18a80374-61a1-4cd0-806d-9578a7e70433\") " pod="openshift-multus/multus-additional-cni-plugins-qjwvr"
Jul 27 18:29:01.055555 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054605    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 18:29:01.055555 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054665    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-645kg\" (UniqueName: \"kubernetes.io/projected/18a80374-61a1-4cd0-806d-9578a7e70433-kube-api-access-645kg\") pod \"multus-additional-cni-plugins-qjwvr\" (UID: \"18a80374-61a1-4cd0-806d-9578a7e70433\") " pod="openshift-multus/multus-additional-cni-plugins-qjwvr"
Jul 27 18:29:01.055555 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054685    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-host-run-multus-certs\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.055555 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054717    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18a80374-61a1-4cd0-806d-9578a7e70433-system-cni-dir\") pod \"multus-additional-cni-plugins-qjwvr\" (UID: \"18a80374-61a1-4cd0-806d-9578a7e70433\") " pod="openshift-multus/multus-additional-cni-plugins-qjwvr"
Jul 27 18:29:01.055555 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054726    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-etc-openvswitch\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.055555 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054780    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-multus-cni-dir\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.055555 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054844    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fa213400-5979-496e-9d82-f6ba6a0f0333-env-overrides\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.055555 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054874    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x8h8\" (UniqueName: \"kubernetes.io/projected/fa213400-5979-496e-9d82-f6ba6a0f0333-kube-api-access-6x8h8\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.055555 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054905    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0806ed23-3336-4327-ae3a-0da1e826f856-host\") pod \"node-ca-4g7qz\" (UID: \"0806ed23-3336-4327-ae3a-0da1e826f856\") " pod="openshift-image-registry/node-ca-4g7qz"
Jul 27 18:29:01.055555 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054913    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-multus-daemon-config\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.055555 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054951    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0806ed23-3336-4327-ae3a-0da1e826f856-host\") pod \"node-ca-4g7qz\" (UID: \"0806ed23-3336-4327-ae3a-0da1e826f856\") " pod="openshift-image-registry/node-ca-4g7qz"
Jul 27 18:29:01.055555 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054953    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0806ed23-3336-4327-ae3a-0da1e826f856-serviceca\") pod \"node-ca-4g7qz\" (UID: \"0806ed23-3336-4327-ae3a-0da1e826f856\") " pod="openshift-image-registry/node-ca-4g7qz"
Jul 27 18:29:01.055555 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054971    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5b7a8e0f-9255-415f-b6fe-123c07105bc3-konnectivity-ca\") pod \"konnectivity-agent-l6rrw\" (UID: \"5b7a8e0f-9255-415f-b6fe-123c07105bc3\") " pod="kube-system/konnectivity-agent-l6rrw"
Jul 27 18:29:01.055555 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.054995    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-node-log\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.056638 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055022    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-host-var-lib-cni-multus\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.056638 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055039    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-multus-conf-dir\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.056638 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055067    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-host-var-lib-cni-multus\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.056638 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055074    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-multus-conf-dir\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.056638 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055096    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-etc-sysconfig\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.056638 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055152    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/18a80374-61a1-4cd0-806d-9578a7e70433-cni-binary-copy\") pod \"multus-additional-cni-plugins-qjwvr\" (UID: \"18a80374-61a1-4cd0-806d-9578a7e70433\") " pod="openshift-multus/multus-additional-cni-plugins-qjwvr"
Jul 27 18:29:01.056638 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055172    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-etc-sysconfig\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.056638 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055188    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-host-cni-netd\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.056638 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055212    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-system-cni-dir\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.056638 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055227    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5b7a8e0f-9255-415f-b6fe-123c07105bc3-agent-certs\") pod \"konnectivity-agent-l6rrw\" (UID: \"5b7a8e0f-9255-415f-b6fe-123c07105bc3\") " pod="kube-system/konnectivity-agent-l6rrw"
Jul 27 18:29:01.056638 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055243    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-etc-kubernetes\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.056638 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055266    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-host-kubelet\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.056638 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055291    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-os-release\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.056638 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055307    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-system-cni-dir\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.056638 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055320    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-etc-sysctl-conf\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.056638 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055346    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b81bd2e0-40bf-41ba-b0ab-2c86fdee9438-registration-dir\") pod \"aws-ebs-csi-driver-node-bxjrv\" (UID: \"b81bd2e0-40bf-41ba-b0ab-2c86fdee9438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv"
Jul 27 18:29:01.056638 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055370    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b81bd2e0-40bf-41ba-b0ab-2c86fdee9438-device-dir\") pod \"aws-ebs-csi-driver-node-bxjrv\" (UID: \"b81bd2e0-40bf-41ba-b0ab-2c86fdee9438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv"
Jul 27 18:29:01.056638 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055379    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-os-release\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.057602 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055390    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4378195f-ed99-42aa-90e6-0fde9f5394f1-tmp-dir\") pod \"node-resolver-cl7cw\" (UID: \"4378195f-ed99-42aa-90e6-0fde9f5394f1\") " pod="openshift-dns/node-resolver-cl7cw"
Jul 27 18:29:01.057602 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055411    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/18a80374-61a1-4cd0-806d-9578a7e70433-os-release\") pod \"multus-additional-cni-plugins-qjwvr\" (UID: \"18a80374-61a1-4cd0-806d-9578a7e70433\") " pod="openshift-multus/multus-additional-cni-plugins-qjwvr"
Jul 27 18:29:01.057602 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055431    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b81bd2e0-40bf-41ba-b0ab-2c86fdee9438-registration-dir\") pod \"aws-ebs-csi-driver-node-bxjrv\" (UID: \"b81bd2e0-40bf-41ba-b0ab-2c86fdee9438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv"
Jul 27 18:29:01.057602 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055321    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-etc-kubernetes\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.057602 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055447    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-etc-sysctl-conf\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.057602 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055438    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/18a80374-61a1-4cd0-806d-9578a7e70433-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qjwvr\" (UID: \"18a80374-61a1-4cd0-806d-9578a7e70433\") " pod="openshift-multus/multus-additional-cni-plugins-qjwvr"
Jul 27 18:29:01.057602 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055477    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0806ed23-3336-4327-ae3a-0da1e826f856-serviceca\") pod \"node-ca-4g7qz\" (UID: \"0806ed23-3336-4327-ae3a-0da1e826f856\") " pod="openshift-image-registry/node-ca-4g7qz"
Jul 27 18:29:01.057602 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055490    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-dbus\") pod \"global-pull-secret-syncer-qrfwk\" (UID: \"662aa4f5-23c0-42d2-8b21-270c4b33e9a5\") " pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:01.057602 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055520    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-host-var-lib-kubelet\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.057602 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055527    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/18a80374-61a1-4cd0-806d-9578a7e70433-os-release\") pod \"multus-additional-cni-plugins-qjwvr\" (UID: \"18a80374-61a1-4cd0-806d-9578a7e70433\") " pod="openshift-multus/multus-additional-cni-plugins-qjwvr"
Jul 27 18:29:01.057602 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055559    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-etc-kubernetes\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.057602 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055580    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-host-var-lib-kubelet\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.057602 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055609    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-etc-modprobe-d\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.057602 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055624    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-etc-kubernetes\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.057602 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055635    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-run\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.057602 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055660    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b81bd2e0-40bf-41ba-b0ab-2c86fdee9438-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bxjrv\" (UID: \"b81bd2e0-40bf-41ba-b0ab-2c86fdee9438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv"
Jul 27 18:29:01.057602 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055677    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b81bd2e0-40bf-41ba-b0ab-2c86fdee9438-device-dir\") pod \"aws-ebs-csi-driver-node-bxjrv\" (UID: \"b81bd2e0-40bf-41ba-b0ab-2c86fdee9438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv"
Jul 27 18:29:01.058436 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055687    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-original-pull-secret\") pod \"global-pull-secret-syncer-qrfwk\" (UID: \"662aa4f5-23c0-42d2-8b21-270c4b33e9a5\") " pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:01.058436 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055716    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-var-lib-openvswitch\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.058436 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055727    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-run\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.058436 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055735    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/18a80374-61a1-4cd0-806d-9578a7e70433-cni-binary-copy\") pod \"multus-additional-cni-plugins-qjwvr\" (UID: \"18a80374-61a1-4cd0-806d-9578a7e70433\") " pod="openshift-multus/multus-additional-cni-plugins-qjwvr"
Jul 27 18:29:01.058436 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055743    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-host-run-ovn-kubernetes\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.058436 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055768    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-cnibin\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.058436 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055790    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-multus-socket-dir-parent\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.058436 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055812    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-etc-modprobe-d\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.058436 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055817    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dff73097-b755-4894-bbd4-8b3a3a69d3a8-tmp\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.058436 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055825    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b81bd2e0-40bf-41ba-b0ab-2c86fdee9438-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bxjrv\" (UID: \"b81bd2e0-40bf-41ba-b0ab-2c86fdee9438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv"
Jul 27 18:29:01.058436 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055858    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b81bd2e0-40bf-41ba-b0ab-2c86fdee9438-socket-dir\") pod \"aws-ebs-csi-driver-node-bxjrv\" (UID: \"b81bd2e0-40bf-41ba-b0ab-2c86fdee9438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv"
Jul 27 18:29:01.058436 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055885    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-run-ovn\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.058436 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055896    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-multus-socket-dir-parent\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.058436 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055903    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4378195f-ed99-42aa-90e6-0fde9f5394f1-tmp-dir\") pod \"node-resolver-cl7cw\" (UID: \"4378195f-ed99-42aa-90e6-0fde9f5394f1\") " pod="openshift-dns/node-resolver-cl7cw"
Jul 27 18:29:01.058436 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055934    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-log-socket\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.058436 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055944    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/18a80374-61a1-4cd0-806d-9578a7e70433-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qjwvr\" (UID: \"18a80374-61a1-4cd0-806d-9578a7e70433\") " pod="openshift-multus/multus-additional-cni-plugins-qjwvr"
Jul 27 18:29:01.058436 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.055992    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b81bd2e0-40bf-41ba-b0ab-2c86fdee9438-socket-dir\") pod \"aws-ebs-csi-driver-node-bxjrv\" (UID: \"b81bd2e0-40bf-41ba-b0ab-2c86fdee9438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv"
Jul 27 18:29:01.058967 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056050    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.058967 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056083    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-var-lib-kubelet\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.058967 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056102    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-cnibin\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.058967 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056107    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8c44c98c-c8de-4a6e-b827-3d56a456f1eb-host-slash\") pod \"iptables-alerter-f8mj4\" (UID: \"8c44c98c-c8de-4a6e-b827-3d56a456f1eb\") " pod="openshift-network-operator/iptables-alerter-f8mj4"
Jul 27 18:29:01.058967 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056139    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-hostroot\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.058967 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056153    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-var-lib-kubelet\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.058967 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056165    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18a80374-61a1-4cd0-806d-9578a7e70433-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qjwvr\" (UID: \"18a80374-61a1-4cd0-806d-9578a7e70433\") " pod="openshift-multus/multus-additional-cni-plugins-qjwvr"
Jul 27 18:29:01.058967 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056196    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-hostroot\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.058967 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056192    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-kubelet-config\") pod \"global-pull-secret-syncer-qrfwk\" (UID: \"662aa4f5-23c0-42d2-8b21-270c4b33e9a5\") " pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:01.058967 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056239    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-run-openvswitch\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.058967 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056262    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-host-cni-bin\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.058967 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056285    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fa213400-5979-496e-9d82-f6ba6a0f0333-ovnkube-config\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.058967 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056323    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7prjs\" (UniqueName: \"kubernetes.io/projected/dff73097-b755-4894-bbd4-8b3a3a69d3a8-kube-api-access-7prjs\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.058967 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056327    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18a80374-61a1-4cd0-806d-9578a7e70433-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qjwvr\" (UID: \"18a80374-61a1-4cd0-806d-9578a7e70433\") " pod="openshift-multus/multus-additional-cni-plugins-qjwvr"
Jul 27 18:29:01.058967 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056351    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8c44c98c-c8de-4a6e-b827-3d56a456f1eb-iptables-alerter-script\") pod \"iptables-alerter-f8mj4\" (UID: \"8c44c98c-c8de-4a6e-b827-3d56a456f1eb\") " pod="openshift-network-operator/iptables-alerter-f8mj4"
Jul 27 18:29:01.058967 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056374    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa213400-5979-496e-9d82-f6ba6a0f0333-ovn-node-metrics-cert\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.058967 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056414    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/18a80374-61a1-4cd0-806d-9578a7e70433-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qjwvr\" (UID: \"18a80374-61a1-4cd0-806d-9578a7e70433\") " pod="openshift-multus/multus-additional-cni-plugins-qjwvr"
Jul 27 18:29:01.059545 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056447    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-host-var-lib-cni-bin\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.059545 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056473    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-etc-systemd\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.059545 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056496    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-host\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.059545 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056515    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-host-var-lib-cni-bin\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.059545 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056536    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b81bd2e0-40bf-41ba-b0ab-2c86fdee9438-etc-selinux\") pod \"aws-ebs-csi-driver-node-bxjrv\" (UID: \"b81bd2e0-40bf-41ba-b0ab-2c86fdee9438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv"
Jul 27 18:29:01.059545 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056566    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6751c8be-8944-44d2-8e25-f3fd6926225e-metrics-certs\") pod \"network-metrics-daemon-qc4qw\" (UID: \"6751c8be-8944-44d2-8e25-f3fd6926225e\") " pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:01.059545 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056571    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-host\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.059545 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056573    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dff73097-b755-4894-bbd4-8b3a3a69d3a8-etc-systemd\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.059545 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056604    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-cni-binary-copy\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.059545 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056694    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b81bd2e0-40bf-41ba-b0ab-2c86fdee9438-etc-selinux\") pod \"aws-ebs-csi-driver-node-bxjrv\" (UID: \"b81bd2e0-40bf-41ba-b0ab-2c86fdee9438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv"
Jul 27 18:29:01.059545 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.056876    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/18a80374-61a1-4cd0-806d-9578a7e70433-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qjwvr\" (UID: \"18a80374-61a1-4cd0-806d-9578a7e70433\") " pod="openshift-multus/multus-additional-cni-plugins-qjwvr"
Jul 27 18:29:01.059545 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.057112    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-cni-binary-copy\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.059545 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.057974    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dff73097-b755-4894-bbd4-8b3a3a69d3a8-etc-tuned\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.059545 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.058220    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5b7a8e0f-9255-415f-b6fe-123c07105bc3-agent-certs\") pod \"konnectivity-agent-l6rrw\" (UID: \"5b7a8e0f-9255-415f-b6fe-123c07105bc3\") " pod="kube-system/konnectivity-agent-l6rrw"
Jul 27 18:29:01.059545 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.058459    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dff73097-b755-4894-bbd4-8b3a3a69d3a8-tmp\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.067515 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.066994    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmxlh\" (UniqueName: \"kubernetes.io/projected/be9de2b2-0f9b-4c07-87f3-048e94a8ab40-kube-api-access-tmxlh\") pod \"multus-z6tg2\" (UID: \"be9de2b2-0f9b-4c07-87f3-048e94a8ab40\") " pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.067515 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.067434    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-645kg\" (UniqueName: \"kubernetes.io/projected/18a80374-61a1-4cd0-806d-9578a7e70433-kube-api-access-645kg\") pod \"multus-additional-cni-plugins-qjwvr\" (UID: \"18a80374-61a1-4cd0-806d-9578a7e70433\") " pod="openshift-multus/multus-additional-cni-plugins-qjwvr"
Jul 27 18:29:01.067934 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.067911    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdgzn\" (UniqueName: \"kubernetes.io/projected/b81bd2e0-40bf-41ba-b0ab-2c86fdee9438-kube-api-access-vdgzn\") pod \"aws-ebs-csi-driver-node-bxjrv\" (UID: \"b81bd2e0-40bf-41ba-b0ab-2c86fdee9438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv"
Jul 27 18:29:01.068035 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.067918    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6wds\" (UniqueName: \"kubernetes.io/projected/4378195f-ed99-42aa-90e6-0fde9f5394f1-kube-api-access-r6wds\") pod \"node-resolver-cl7cw\" (UID: \"4378195f-ed99-42aa-90e6-0fde9f5394f1\") " pod="openshift-dns/node-resolver-cl7cw"
Jul 27 18:29:01.068138 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.068115    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4h2n\" (UniqueName: \"kubernetes.io/projected/0806ed23-3336-4327-ae3a-0da1e826f856-kube-api-access-w4h2n\") pod \"node-ca-4g7qz\" (UID: \"0806ed23-3336-4327-ae3a-0da1e826f856\") " pod="openshift-image-registry/node-ca-4g7qz"
Jul 27 18:29:01.068260 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.068241    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7prjs\" (UniqueName: \"kubernetes.io/projected/dff73097-b755-4894-bbd4-8b3a3a69d3a8-kube-api-access-7prjs\") pod \"tuned-zqkqx\" (UID: \"dff73097-b755-4894-bbd4-8b3a3a69d3a8\") " pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.156995 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.156963    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-host-run-netns\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.156995 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157000    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcsl5\" (UniqueName: \"kubernetes.io/projected/8c44c98c-c8de-4a6e-b827-3d56a456f1eb-kube-api-access-pcsl5\") pod \"iptables-alerter-f8mj4\" (UID: \"8c44c98c-c8de-4a6e-b827-3d56a456f1eb\") " pod="openshift-network-operator/iptables-alerter-f8mj4"
Jul 27 18:29:01.157400 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157031    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fa213400-5979-496e-9d82-f6ba6a0f0333-ovnkube-script-lib\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.157400 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157055    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vnls\" (UniqueName: \"kubernetes.io/projected/6751c8be-8944-44d2-8e25-f3fd6926225e-kube-api-access-2vnls\") pod \"network-metrics-daemon-qc4qw\" (UID: \"6751c8be-8944-44d2-8e25-f3fd6926225e\") " pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:01.157400 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157073    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dftwx\" (UniqueName: \"kubernetes.io/projected/53874b39-22ec-4554-ab95-643a49ee8699-kube-api-access-dftwx\") pod \"network-check-target-jpq2j\" (UID: \"53874b39-22ec-4554-ab95-643a49ee8699\") " pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:01.157400 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157088    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-systemd-units\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.157400 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157089    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-host-run-netns\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.157400 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157104    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-host-slash\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.157400 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157183    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-run-systemd\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.157400 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157197    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-systemd-units\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.157400 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157251    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-etc-openvswitch\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.157400 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157214    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-etc-openvswitch\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.157400 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157269    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-run-systemd\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.157400 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157293    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fa213400-5979-496e-9d82-f6ba6a0f0333-env-overrides\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.157400 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157318    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x8h8\" (UniqueName: \"kubernetes.io/projected/fa213400-5979-496e-9d82-f6ba6a0f0333-kube-api-access-6x8h8\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.157400 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157342    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-host-slash\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.157400 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157365    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-node-log\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.157400 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157401    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-node-log\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.157400 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157408    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-host-cni-netd\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.158242 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157447    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-host-kubelet\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.158242 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157479    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-dbus\") pod \"global-pull-secret-syncer-qrfwk\" (UID: \"662aa4f5-23c0-42d2-8b21-270c4b33e9a5\") " pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:01.158242 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157487    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-host-cni-netd\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.158242 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157509    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-original-pull-secret\") pod \"global-pull-secret-syncer-qrfwk\" (UID: \"662aa4f5-23c0-42d2-8b21-270c4b33e9a5\") " pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:01.158242 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157535    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-var-lib-openvswitch\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.158242 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157564    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-host-run-ovn-kubernetes\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.158242 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157542    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-host-kubelet\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.158242 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157621    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-var-lib-openvswitch\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.158242 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:01.157640    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 18:29:01.158242 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157656    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-run-ovn\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.158242 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157696    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-host-run-ovn-kubernetes\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.158242 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157683    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-log-socket\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.158242 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157643    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-dbus\") pod \"global-pull-secret-syncer-qrfwk\" (UID: \"662aa4f5-23c0-42d2-8b21-270c4b33e9a5\") " pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:01.158242 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:01.157732    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-original-pull-secret podName:662aa4f5-23c0-42d2-8b21-270c4b33e9a5 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:01.657698789 +0000 UTC m=+2.091956393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-original-pull-secret") pod "global-pull-secret-syncer-qrfwk" (UID: "662aa4f5-23c0-42d2-8b21-270c4b33e9a5") : object "kube-system"/"original-pull-secret" not registered
Jul 27 18:29:01.158242 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157757    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-run-ovn\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.158242 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157773    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fa213400-5979-496e-9d82-f6ba6a0f0333-ovnkube-script-lib\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.158242 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157780    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.158900 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157758    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-log-socket\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.158900 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157803    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fa213400-5979-496e-9d82-f6ba6a0f0333-env-overrides\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.158900 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157820    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8c44c98c-c8de-4a6e-b827-3d56a456f1eb-host-slash\") pod \"iptables-alerter-f8mj4\" (UID: \"8c44c98c-c8de-4a6e-b827-3d56a456f1eb\") " pod="openshift-network-operator/iptables-alerter-f8mj4"
Jul 27 18:29:01.158900 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157866    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.158900 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157874    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8c44c98c-c8de-4a6e-b827-3d56a456f1eb-host-slash\") pod \"iptables-alerter-f8mj4\" (UID: \"8c44c98c-c8de-4a6e-b827-3d56a456f1eb\") " pod="openshift-network-operator/iptables-alerter-f8mj4"
Jul 27 18:29:01.158900 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157890    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-kubelet-config\") pod \"global-pull-secret-syncer-qrfwk\" (UID: \"662aa4f5-23c0-42d2-8b21-270c4b33e9a5\") " pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:01.158900 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157918    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-run-openvswitch\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.158900 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157938    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-host-cni-bin\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.158900 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157955    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fa213400-5979-496e-9d82-f6ba6a0f0333-ovnkube-config\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.158900 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157967    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-kubelet-config\") pod \"global-pull-secret-syncer-qrfwk\" (UID: \"662aa4f5-23c0-42d2-8b21-270c4b33e9a5\") " pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:01.158900 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.157987    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-run-openvswitch\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.158900 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.158016    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa213400-5979-496e-9d82-f6ba6a0f0333-host-cni-bin\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.158900 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.158154    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8c44c98c-c8de-4a6e-b827-3d56a456f1eb-iptables-alerter-script\") pod \"iptables-alerter-f8mj4\" (UID: \"8c44c98c-c8de-4a6e-b827-3d56a456f1eb\") " pod="openshift-network-operator/iptables-alerter-f8mj4"
Jul 27 18:29:01.158900 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.158182    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa213400-5979-496e-9d82-f6ba6a0f0333-ovn-node-metrics-cert\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.158900 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.158216    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6751c8be-8944-44d2-8e25-f3fd6926225e-metrics-certs\") pod \"network-metrics-daemon-qc4qw\" (UID: \"6751c8be-8944-44d2-8e25-f3fd6926225e\") " pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:01.158900 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:01.158314    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 18:29:01.158900 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:01.158377    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6751c8be-8944-44d2-8e25-f3fd6926225e-metrics-certs podName:6751c8be-8944-44d2-8e25-f3fd6926225e nodeName:}" failed. No retries permitted until 2026-07-27 18:29:01.658357238 +0000 UTC m=+2.092614834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6751c8be-8944-44d2-8e25-f3fd6926225e-metrics-certs") pod "network-metrics-daemon-qc4qw" (UID: "6751c8be-8944-44d2-8e25-f3fd6926225e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 18:29:01.159567 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.158588    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fa213400-5979-496e-9d82-f6ba6a0f0333-ovnkube-config\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.159567 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.158615    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8c44c98c-c8de-4a6e-b827-3d56a456f1eb-iptables-alerter-script\") pod \"iptables-alerter-f8mj4\" (UID: \"8c44c98c-c8de-4a6e-b827-3d56a456f1eb\") " pod="openshift-network-operator/iptables-alerter-f8mj4"
Jul 27 18:29:01.160756 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.160735    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa213400-5979-496e-9d82-f6ba6a0f0333-ovn-node-metrics-cert\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.170995 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:01.170970    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 18:29:01.170995 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:01.170994    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 18:29:01.171185 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:01.171006    2566 projected.go:196] Error preparing data for projected volume kube-api-access-dftwx for pod openshift-network-diagnostics/network-check-target-jpq2j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 18:29:01.171185 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:01.171070    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53874b39-22ec-4554-ab95-643a49ee8699-kube-api-access-dftwx podName:53874b39-22ec-4554-ab95-643a49ee8699 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:01.671056427 +0000 UTC m=+2.105314018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dftwx" (UniqueName: "kubernetes.io/projected/53874b39-22ec-4554-ab95-643a49ee8699-kube-api-access-dftwx") pod "network-check-target-jpq2j" (UID: "53874b39-22ec-4554-ab95-643a49ee8699") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 18:29:01.171695 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.171643    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vnls\" (UniqueName: \"kubernetes.io/projected/6751c8be-8944-44d2-8e25-f3fd6926225e-kube-api-access-2vnls\") pod \"network-metrics-daemon-qc4qw\" (UID: \"6751c8be-8944-44d2-8e25-f3fd6926225e\") " pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:01.172028 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.172008    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcsl5\" (UniqueName: \"kubernetes.io/projected/8c44c98c-c8de-4a6e-b827-3d56a456f1eb-kube-api-access-pcsl5\") pod \"iptables-alerter-f8mj4\" (UID: \"8c44c98c-c8de-4a6e-b827-3d56a456f1eb\") " pod="openshift-network-operator/iptables-alerter-f8mj4"
Jul 27 18:29:01.174250 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.173872    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x8h8\" (UniqueName: \"kubernetes.io/projected/fa213400-5979-496e-9d82-f6ba6a0f0333-kube-api-access-6x8h8\") pod \"ovnkube-node-tdb9t\" (UID: \"fa213400-5979-496e-9d82-f6ba6a0f0333\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.237392 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:29:01.237352    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20c1cf26d726b4ecc2d495ed14b7a50.slice/crio-6fea475223d2dca54295fcd0170b5cd345d2e0ab7eeba084b1669990587999e4 WatchSource:0}: Error finding container 6fea475223d2dca54295fcd0170b5cd345d2e0ab7eeba084b1669990587999e4: Status 404 returned error can't find the container with id 6fea475223d2dca54295fcd0170b5cd345d2e0ab7eeba084b1669990587999e4
Jul 27 18:29:01.237747 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:29:01.237728    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd8d2439d53474a3dfd84498491b9ab.slice/crio-21853b497c0eaeb81f90543a3cb572b6033eff73d4f8f056e7170cea7bcce700 WatchSource:0}: Error finding container 21853b497c0eaeb81f90543a3cb572b6033eff73d4f8f056e7170cea7bcce700: Status 404 returned error can't find the container with id 21853b497c0eaeb81f90543a3cb572b6033eff73d4f8f056e7170cea7bcce700
Jul 27 18:29:01.244030 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.244008    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 18:29:01.267132 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.267089    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv"
Jul 27 18:29:01.276693 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:29:01.276663    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb81bd2e0_40bf_41ba_b0ab_2c86fdee9438.slice/crio-906f78869d96e0d3b7b896b7f8e9e16d209cc8eb3d6bf0d5a88972e4ef4eb782 WatchSource:0}: Error finding container 906f78869d96e0d3b7b896b7f8e9e16d209cc8eb3d6bf0d5a88972e4ef4eb782: Status 404 returned error can't find the container with id 906f78869d96e0d3b7b896b7f8e9e16d209cc8eb3d6bf0d5a88972e4ef4eb782
Jul 27 18:29:01.288294 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.288270    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cl7cw"
Jul 27 18:29:01.296248 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:29:01.296219    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4378195f_ed99_42aa_90e6_0fde9f5394f1.slice/crio-f82b826ddd409a06e5589d0fb66264c45f0dcac5789dd58e7de37823f5736c49 WatchSource:0}: Error finding container f82b826ddd409a06e5589d0fb66264c45f0dcac5789dd58e7de37823f5736c49: Status 404 returned error can't find the container with id f82b826ddd409a06e5589d0fb66264c45f0dcac5789dd58e7de37823f5736c49
Jul 27 18:29:01.308344 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.308320    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4g7qz"
Jul 27 18:29:01.316272 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:29:01.316246    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0806ed23_3336_4327_ae3a_0da1e826f856.slice/crio-a6e76e1bc6a91e54d5d6b180bf8e8b66624c8687262170f33ec7d686a3cd8065 WatchSource:0}: Error finding container a6e76e1bc6a91e54d5d6b180bf8e8b66624c8687262170f33ec7d686a3cd8065: Status 404 returned error can't find the container with id a6e76e1bc6a91e54d5d6b180bf8e8b66624c8687262170f33ec7d686a3cd8065
Jul 27 18:29:01.324014 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.323994    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qjwvr"
Jul 27 18:29:01.331939 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:29:01.331893    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a80374_61a1_4cd0_806d_9578a7e70433.slice/crio-723c4164e82a042b11e6b0f6c1237922d7fc21a89d8d9304322984e83391a867 WatchSource:0}: Error finding container 723c4164e82a042b11e6b0f6c1237922d7fc21a89d8d9304322984e83391a867: Status 404 returned error can't find the container with id 723c4164e82a042b11e6b0f6c1237922d7fc21a89d8d9304322984e83391a867
Jul 27 18:29:01.332712 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.332683    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z6tg2"
Jul 27 18:29:01.340935 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:29:01.340913    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9de2b2_0f9b_4c07_87f3_048e94a8ab40.slice/crio-4e4ea3d5defdd3f5f2c9f1df78184948c5e2caaab00ee66e1181f90d1bb044c1 WatchSource:0}: Error finding container 4e4ea3d5defdd3f5f2c9f1df78184948c5e2caaab00ee66e1181f90d1bb044c1: Status 404 returned error can't find the container with id 4e4ea3d5defdd3f5f2c9f1df78184948c5e2caaab00ee66e1181f90d1bb044c1
Jul 27 18:29:01.344720 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.344702    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l6rrw"
Jul 27 18:29:01.352435 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:29:01.352412    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b7a8e0f_9255_415f_b6fe_123c07105bc3.slice/crio-47f0f2777b2df389c0e099e10dbf82e1554659415d9e09db543c24db6de685f3 WatchSource:0}: Error finding container 47f0f2777b2df389c0e099e10dbf82e1554659415d9e09db543c24db6de685f3: Status 404 returned error can't find the container with id 47f0f2777b2df389c0e099e10dbf82e1554659415d9e09db543c24db6de685f3
Jul 27 18:29:01.365073 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.365044    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zqkqx"
Jul 27 18:29:01.373028 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:29:01.373006    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff73097_b755_4894_bbd4_8b3a3a69d3a8.slice/crio-4918910fe6f908a9847b0e25bbd2c4a4591f8398e86b66b8d9d16c6a2333824a WatchSource:0}: Error finding container 4918910fe6f908a9847b0e25bbd2c4a4591f8398e86b66b8d9d16c6a2333824a: Status 404 returned error can't find the container with id 4918910fe6f908a9847b0e25bbd2c4a4591f8398e86b66b8d9d16c6a2333824a
Jul 27 18:29:01.393648 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.393619    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f8mj4"
Jul 27 18:29:01.399824 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.399799    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:01.403416 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:29:01.403387    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c44c98c_c8de_4a6e_b827_3d56a456f1eb.slice/crio-290c54d72c93806db07b338c53b632d5fc13ab269d3ee79493f01847cf2324e9 WatchSource:0}: Error finding container 290c54d72c93806db07b338c53b632d5fc13ab269d3ee79493f01847cf2324e9: Status 404 returned error can't find the container with id 290c54d72c93806db07b338c53b632d5fc13ab269d3ee79493f01847cf2324e9
Jul 27 18:29:01.408361 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:29:01.408339    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa213400_5979_496e_9d82_f6ba6a0f0333.slice/crio-e15cb3601d90b1c46a00d7f642a6cdd8ad1c69ca5a2882c3941075cc3443e8a7 WatchSource:0}: Error finding container e15cb3601d90b1c46a00d7f642a6cdd8ad1c69ca5a2882c3941075cc3443e8a7: Status 404 returned error can't find the container with id e15cb3601d90b1c46a00d7f642a6cdd8ad1c69ca5a2882c3941075cc3443e8a7
Jul 27 18:29:01.410167 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:01.410148    2566 kuberuntime_manager.go:1459] "Unhandled Error" err="container ovn-controller start failed in pod ovnkube-node-tdb9t_openshift-ovn-kubernetes(fa213400-5979-496e-9d82-f6ba6a0f0333): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Jul 27 18:29:01.411582 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:01.411565    2566 kuberuntime_manager.go:1459] "Unhandled Error" err="container kube-rbac-proxy-node start failed in pod ovnkube-node-tdb9t_openshift-ovn-kubernetes(fa213400-5979-496e-9d82-f6ba6a0f0333): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Jul 27 18:29:01.414738 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:01.414690    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t" podUID="fa213400-5979-496e-9d82-f6ba6a0f0333"
Jul 27 18:29:01.662450 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.662418    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-original-pull-secret\") pod \"global-pull-secret-syncer-qrfwk\" (UID: \"662aa4f5-23c0-42d2-8b21-270c4b33e9a5\") " pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:01.662637 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.662473    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6751c8be-8944-44d2-8e25-f3fd6926225e-metrics-certs\") pod \"network-metrics-daemon-qc4qw\" (UID: \"6751c8be-8944-44d2-8e25-f3fd6926225e\") " pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:01.662637 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:01.662572    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 18:29:01.662637 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:01.662577    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 18:29:01.662637 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:01.662630    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6751c8be-8944-44d2-8e25-f3fd6926225e-metrics-certs podName:6751c8be-8944-44d2-8e25-f3fd6926225e nodeName:}" failed. No retries permitted until 2026-07-27 18:29:02.662617366 +0000 UTC m=+3.096874956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6751c8be-8944-44d2-8e25-f3fd6926225e-metrics-certs") pod "network-metrics-daemon-qc4qw" (UID: "6751c8be-8944-44d2-8e25-f3fd6926225e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 18:29:01.662778 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:01.662644    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-original-pull-secret podName:662aa4f5-23c0-42d2-8b21-270c4b33e9a5 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:02.662637792 +0000 UTC m=+3.096895383 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-original-pull-secret") pod "global-pull-secret-syncer-qrfwk" (UID: "662aa4f5-23c0-42d2-8b21-270c4b33e9a5") : object "kube-system"/"original-pull-secret" not registered
Jul 27 18:29:01.763135 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.763100    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dftwx\" (UniqueName: \"kubernetes.io/projected/53874b39-22ec-4554-ab95-643a49ee8699-kube-api-access-dftwx\") pod \"network-check-target-jpq2j\" (UID: \"53874b39-22ec-4554-ab95-643a49ee8699\") " pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:01.763386 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:01.763341    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 18:29:01.763386 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:01.763369    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 18:29:01.763386 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:01.763385    2566 projected.go:196] Error preparing data for projected volume kube-api-access-dftwx for pod openshift-network-diagnostics/network-check-target-jpq2j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 18:29:01.763549 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:01.763447    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53874b39-22ec-4554-ab95-643a49ee8699-kube-api-access-dftwx podName:53874b39-22ec-4554-ab95-643a49ee8699 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:02.763428441 +0000 UTC m=+3.197686047 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dftwx" (UniqueName: "kubernetes.io/projected/53874b39-22ec-4554-ab95-643a49ee8699-kube-api-access-dftwx") pod "network-check-target-jpq2j" (UID: "53874b39-22ec-4554-ab95-643a49ee8699") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 18:29:01.995773 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.995655    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 18:24:01 +0000 UTC" deadline="2028-04-06 23:29:13.81853468 +0000 UTC"
Jul 27 18:29:01.995773 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:01.995718    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14861h0m11.822821212s"
Jul 27 18:29:02.072388 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:02.072355    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 18:29:02.130212 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:02.130132    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4g7qz" event={"ID":"0806ed23-3336-4327-ae3a-0da1e826f856","Type":"ContainerStarted","Data":"a6e76e1bc6a91e54d5d6b180bf8e8b66624c8687262170f33ec7d686a3cd8065"}
Jul 27 18:29:02.134041 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:02.134010    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cl7cw" event={"ID":"4378195f-ed99-42aa-90e6-0fde9f5394f1","Type":"ContainerStarted","Data":"f82b826ddd409a06e5589d0fb66264c45f0dcac5789dd58e7de37823f5736c49"}
Jul 27 18:29:02.135737 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:02.135708    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-196.ec2.internal" event={"ID":"7cd8d2439d53474a3dfd84498491b9ab","Type":"ContainerStarted","Data":"21853b497c0eaeb81f90543a3cb572b6033eff73d4f8f056e7170cea7bcce700"}
Jul 27 18:29:02.137870 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:02.137844    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f8mj4" event={"ID":"8c44c98c-c8de-4a6e-b827-3d56a456f1eb","Type":"ContainerStarted","Data":"290c54d72c93806db07b338c53b632d5fc13ab269d3ee79493f01847cf2324e9"}
Jul 27 18:29:02.139271 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:02.139244    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zqkqx" event={"ID":"dff73097-b755-4894-bbd4-8b3a3a69d3a8","Type":"ContainerStarted","Data":"4918910fe6f908a9847b0e25bbd2c4a4591f8398e86b66b8d9d16c6a2333824a"}
Jul 27 18:29:02.142720 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:02.142693    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z6tg2" event={"ID":"be9de2b2-0f9b-4c07-87f3-048e94a8ab40","Type":"ContainerStarted","Data":"4e4ea3d5defdd3f5f2c9f1df78184948c5e2caaab00ee66e1181f90d1bb044c1"}
Jul 27 18:29:02.149397 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:02.149338    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv" event={"ID":"b81bd2e0-40bf-41ba-b0ab-2c86fdee9438","Type":"ContainerStarted","Data":"906f78869d96e0d3b7b896b7f8e9e16d209cc8eb3d6bf0d5a88972e4ef4eb782"}
Jul 27 18:29:02.150847 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:02.150774    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-196.ec2.internal" event={"ID":"b20c1cf26d726b4ecc2d495ed14b7a50","Type":"ContainerStarted","Data":"6fea475223d2dca54295fcd0170b5cd345d2e0ab7eeba084b1669990587999e4"}
Jul 27 18:29:02.153247 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:02.152357    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t" event={"ID":"fa213400-5979-496e-9d82-f6ba6a0f0333","Type":"ContainerStarted","Data":"e15cb3601d90b1c46a00d7f642a6cdd8ad1c69ca5a2882c3941075cc3443e8a7"}
Jul 27 18:29:02.162724 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:02.162694    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l6rrw" event={"ID":"5b7a8e0f-9255-415f-b6fe-123c07105bc3","Type":"ContainerStarted","Data":"47f0f2777b2df389c0e099e10dbf82e1554659415d9e09db543c24db6de685f3"}
Jul 27 18:29:02.173813 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:02.173734    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t" podUID="fa213400-5979-496e-9d82-f6ba6a0f0333"
Jul 27 18:29:02.173945 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:02.173917    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qjwvr" event={"ID":"18a80374-61a1-4cd0-806d-9578a7e70433","Type":"ContainerStarted","Data":"723c4164e82a042b11e6b0f6c1237922d7fc21a89d8d9304322984e83391a867"}
Jul 27 18:29:02.671595 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:02.671532    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-original-pull-secret\") pod \"global-pull-secret-syncer-qrfwk\" (UID: \"662aa4f5-23c0-42d2-8b21-270c4b33e9a5\") " pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:02.671595 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:02.671603    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6751c8be-8944-44d2-8e25-f3fd6926225e-metrics-certs\") pod \"network-metrics-daemon-qc4qw\" (UID: \"6751c8be-8944-44d2-8e25-f3fd6926225e\") " pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:02.671849 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:02.671754    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 18:29:02.671849 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:02.671835    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6751c8be-8944-44d2-8e25-f3fd6926225e-metrics-certs podName:6751c8be-8944-44d2-8e25-f3fd6926225e nodeName:}" failed. No retries permitted until 2026-07-27 18:29:04.671811228 +0000 UTC m=+5.106068835 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6751c8be-8944-44d2-8e25-f3fd6926225e-metrics-certs") pod "network-metrics-daemon-qc4qw" (UID: "6751c8be-8944-44d2-8e25-f3fd6926225e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 18:29:02.671967 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:02.671907    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 18:29:02.671967 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:02.671941    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-original-pull-secret podName:662aa4f5-23c0-42d2-8b21-270c4b33e9a5 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:04.671930343 +0000 UTC m=+5.106187938 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-original-pull-secret") pod "global-pull-secret-syncer-qrfwk" (UID: "662aa4f5-23c0-42d2-8b21-270c4b33e9a5") : object "kube-system"/"original-pull-secret" not registered
Jul 27 18:29:02.772549 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:02.772434    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dftwx\" (UniqueName: \"kubernetes.io/projected/53874b39-22ec-4554-ab95-643a49ee8699-kube-api-access-dftwx\") pod \"network-check-target-jpq2j\" (UID: \"53874b39-22ec-4554-ab95-643a49ee8699\") " pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:02.772727 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:02.772612    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 18:29:02.772727 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:02.772633    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 18:29:02.772727 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:02.772647    2566 projected.go:196] Error preparing data for projected volume kube-api-access-dftwx for pod openshift-network-diagnostics/network-check-target-jpq2j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 18:29:02.772727 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:02.772712    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53874b39-22ec-4554-ab95-643a49ee8699-kube-api-access-dftwx podName:53874b39-22ec-4554-ab95-643a49ee8699 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:04.772692979 +0000 UTC m=+5.206950586 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dftwx" (UniqueName: "kubernetes.io/projected/53874b39-22ec-4554-ab95-643a49ee8699-kube-api-access-dftwx") pod "network-check-target-jpq2j" (UID: "53874b39-22ec-4554-ab95-643a49ee8699") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 18:29:02.996816 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:02.996718    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 18:24:01 +0000 UTC" deadline="2028-01-14 06:16:16.221859067 +0000 UTC"
Jul 27 18:29:02.996816 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:02.996790    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12851h47m13.225075019s"
Jul 27 18:29:03.117856 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:03.117797    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:03.118050 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:03.117926    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:03.118376 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:03.118356    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:03.118479 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:03.118463    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:03.118538 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:03.118527    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:03.118635 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:03.118608    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:03.831687 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:03.831645    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 18:29:04.689869 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:04.689165    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-original-pull-secret\") pod \"global-pull-secret-syncer-qrfwk\" (UID: \"662aa4f5-23c0-42d2-8b21-270c4b33e9a5\") " pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:04.689869 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:04.689221    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6751c8be-8944-44d2-8e25-f3fd6926225e-metrics-certs\") pod \"network-metrics-daemon-qc4qw\" (UID: \"6751c8be-8944-44d2-8e25-f3fd6926225e\") " pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:04.689869 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:04.689364    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 18:29:04.689869 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:04.689422    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6751c8be-8944-44d2-8e25-f3fd6926225e-metrics-certs podName:6751c8be-8944-44d2-8e25-f3fd6926225e nodeName:}" failed. No retries permitted until 2026-07-27 18:29:08.689404276 +0000 UTC m=+9.123661880 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6751c8be-8944-44d2-8e25-f3fd6926225e-metrics-certs") pod "network-metrics-daemon-qc4qw" (UID: "6751c8be-8944-44d2-8e25-f3fd6926225e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 18:29:04.689869 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:04.689786    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 18:29:04.689869 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:04.689827    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-original-pull-secret podName:662aa4f5-23c0-42d2-8b21-270c4b33e9a5 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:08.689817096 +0000 UTC m=+9.124074684 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-original-pull-secret") pod "global-pull-secret-syncer-qrfwk" (UID: "662aa4f5-23c0-42d2-8b21-270c4b33e9a5") : object "kube-system"/"original-pull-secret" not registered
Jul 27 18:29:04.789803 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:04.789772    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dftwx\" (UniqueName: \"kubernetes.io/projected/53874b39-22ec-4554-ab95-643a49ee8699-kube-api-access-dftwx\") pod \"network-check-target-jpq2j\" (UID: \"53874b39-22ec-4554-ab95-643a49ee8699\") " pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:04.790013 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:04.789990    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 18:29:04.790177 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:04.790023    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 18:29:04.790177 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:04.790036    2566 projected.go:196] Error preparing data for projected volume kube-api-access-dftwx for pod openshift-network-diagnostics/network-check-target-jpq2j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 18:29:04.790177 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:04.790106    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53874b39-22ec-4554-ab95-643a49ee8699-kube-api-access-dftwx podName:53874b39-22ec-4554-ab95-643a49ee8699 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:08.790086501 +0000 UTC m=+9.224344111 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dftwx" (UniqueName: "kubernetes.io/projected/53874b39-22ec-4554-ab95-643a49ee8699-kube-api-access-dftwx") pod "network-check-target-jpq2j" (UID: "53874b39-22ec-4554-ab95-643a49ee8699") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 18:29:05.117807 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:05.117769    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:05.117980 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:05.117914    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:05.117980 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:05.117782    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:05.118088 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:05.117988    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:05.118088 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:05.117782    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:05.118088 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:05.118056    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:07.117934 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:07.117896    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:07.118451 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:07.118022    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:07.118451 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:07.118440    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:07.118591 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:07.118568    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:07.118654 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:07.118631    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:07.118712 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:07.118697    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:08.722281 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:08.721492    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-original-pull-secret\") pod \"global-pull-secret-syncer-qrfwk\" (UID: \"662aa4f5-23c0-42d2-8b21-270c4b33e9a5\") " pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:08.722281 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:08.721597    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6751c8be-8944-44d2-8e25-f3fd6926225e-metrics-certs\") pod \"network-metrics-daemon-qc4qw\" (UID: \"6751c8be-8944-44d2-8e25-f3fd6926225e\") " pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:08.722281 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:08.721734    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 18:29:08.722281 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:08.721788    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6751c8be-8944-44d2-8e25-f3fd6926225e-metrics-certs podName:6751c8be-8944-44d2-8e25-f3fd6926225e nodeName:}" failed. No retries permitted until 2026-07-27 18:29:16.721772835 +0000 UTC m=+17.156030423 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6751c8be-8944-44d2-8e25-f3fd6926225e-metrics-certs") pod "network-metrics-daemon-qc4qw" (UID: "6751c8be-8944-44d2-8e25-f3fd6926225e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 18:29:08.722281 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:08.722154    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 18:29:08.722281 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:08.722207    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-original-pull-secret podName:662aa4f5-23c0-42d2-8b21-270c4b33e9a5 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:16.72219146 +0000 UTC m=+17.156449066 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-original-pull-secret") pod "global-pull-secret-syncer-qrfwk" (UID: "662aa4f5-23c0-42d2-8b21-270c4b33e9a5") : object "kube-system"/"original-pull-secret" not registered
Jul 27 18:29:08.822970 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:08.822934    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dftwx\" (UniqueName: \"kubernetes.io/projected/53874b39-22ec-4554-ab95-643a49ee8699-kube-api-access-dftwx\") pod \"network-check-target-jpq2j\" (UID: \"53874b39-22ec-4554-ab95-643a49ee8699\") " pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:08.823132 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:08.823067    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 18:29:08.823132 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:08.823083    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 18:29:08.823132 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:08.823094    2566 projected.go:196] Error preparing data for projected volume kube-api-access-dftwx for pod openshift-network-diagnostics/network-check-target-jpq2j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 18:29:08.823245 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:08.823160    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53874b39-22ec-4554-ab95-643a49ee8699-kube-api-access-dftwx podName:53874b39-22ec-4554-ab95-643a49ee8699 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:16.823142464 +0000 UTC m=+17.257400073 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dftwx" (UniqueName: "kubernetes.io/projected/53874b39-22ec-4554-ab95-643a49ee8699-kube-api-access-dftwx") pod "network-check-target-jpq2j" (UID: "53874b39-22ec-4554-ab95-643a49ee8699") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 18:29:09.117648 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:09.117619    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:09.117844 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:09.117728    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:09.118080 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:09.118063    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:09.118199 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:09.118178    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:09.118283 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:09.118235    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:09.118339 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:09.118299    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:11.118290 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:11.118247    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:11.118734 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:11.118247    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:11.118734 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:11.118378    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:11.118734 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:11.118247    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:11.118734 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:11.118476    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:11.118734 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:11.118564    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:13.117410 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:13.117369    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:13.117870 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:13.117422    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:13.117870 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:13.117494    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:13.117870 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:13.117511    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:13.117870 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:13.117589    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:13.117870 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:13.117676    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:15.118357 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:15.118182    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:15.118725 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:15.118190    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:15.118725 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:15.118433    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:15.118725 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:15.118512    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:15.118725 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:15.118198    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:15.118725 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:15.118590    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:15.200154 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:15.200038    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-196.ec2.internal" event={"ID":"7cd8d2439d53474a3dfd84498491b9ab","Type":"ContainerStarted","Data":"f37d2b4d7771d12f32a7e0ae2b25ba44bc34304c1c7f69d5ce11ca1f0b132d24"}
Jul 27 18:29:15.201724 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:15.201694    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zqkqx" event={"ID":"dff73097-b755-4894-bbd4-8b3a3a69d3a8","Type":"ContainerStarted","Data":"1cbfdba94c302aa2062eb0943d0390eb3ca9c68c221d37ba4894a6ac382c3c45"}
Jul 27 18:29:15.203160 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:15.203110    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z6tg2" event={"ID":"be9de2b2-0f9b-4c07-87f3-048e94a8ab40","Type":"ContainerStarted","Data":"d1c19608dca2fb9c2880ed454ab120bafd847d406811932d09ed3d7d5e484317"}
Jul 27 18:29:15.217994 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:15.217940    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-196.ec2.internal" podStartSLOduration=14.217921875 podStartE2EDuration="14.217921875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:15.217777936 +0000 UTC m=+15.652035549" watchObservedRunningTime="2026-07-27 18:29:15.217921875 +0000 UTC m=+15.652179478"
Jul 27 18:29:15.244695 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:15.244633    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-z6tg2" podStartSLOduration=1.771228845 podStartE2EDuration="15.244616857s" totalImagesPullingTime="13.473388012s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:29:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:15.241404587 +0000 UTC m=+15.675662198" watchObservedRunningTime="2026-07-27 18:29:15.244616857 +0000 UTC m=+15.678874468"
Jul 27 18:29:15.286985 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:15.286939    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zqkqx" podStartSLOduration=1.880931323 podStartE2EDuration="15.286925223s" totalImagesPullingTime="13.4059939s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:29:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:15.28624989 +0000 UTC m=+15.720507501" watchObservedRunningTime="2026-07-27 18:29:15.286925223 +0000 UTC m=+15.721182880"
Jul 27 18:29:16.205841 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:16.205647    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv" event={"ID":"b81bd2e0-40bf-41ba-b0ab-2c86fdee9438","Type":"ContainerStarted","Data":"24f8c9643d3d24f555a19e7feba22f6caa29a053c9cc857bb5685cf600ba5337"}
Jul 27 18:29:16.207140 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:16.207092    2566 generic.go:356] "Generic (PLEG): container finished" podID="b20c1cf26d726b4ecc2d495ed14b7a50" containerID="7d1a65e25bc608bdc2e9bb1cb1fd4fbd924dabe538903bee6615312bd77283af" exitCode=0
Jul 27 18:29:16.207245 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:16.207165    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-196.ec2.internal" event={"ID":"b20c1cf26d726b4ecc2d495ed14b7a50","Type":"ContainerDied","Data":"7d1a65e25bc608bdc2e9bb1cb1fd4fbd924dabe538903bee6615312bd77283af"}
Jul 27 18:29:16.208410 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:16.208388    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l6rrw" event={"ID":"5b7a8e0f-9255-415f-b6fe-123c07105bc3","Type":"ContainerStarted","Data":"7ee7bdb0270cd8c4ac3b3a9d969daf40312dc8bb40bbecbc6836439389c6d237"}
Jul 27 18:29:16.209766 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:16.209742    2566 generic.go:356] "Generic (PLEG): container finished" podID="18a80374-61a1-4cd0-806d-9578a7e70433" containerID="65eafbbfc6538df24ac6fbf20491246aa0f80339084de9e18d1146ce69447ff5" exitCode=0
Jul 27 18:29:16.209863 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:16.209827    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qjwvr" event={"ID":"18a80374-61a1-4cd0-806d-9578a7e70433","Type":"ContainerDied","Data":"65eafbbfc6538df24ac6fbf20491246aa0f80339084de9e18d1146ce69447ff5"}
Jul 27 18:29:16.211142 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:16.211103    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4g7qz" event={"ID":"0806ed23-3336-4327-ae3a-0da1e826f856","Type":"ContainerStarted","Data":"077d9b315b3c5a0f7893dd0d532b23a82b38022456e66d1e19b781f9c497e79a"}
Jul 27 18:29:16.212359 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:16.212337    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cl7cw" event={"ID":"4378195f-ed99-42aa-90e6-0fde9f5394f1","Type":"ContainerStarted","Data":"7014dd1cd348c201193d06136bc05d13b272c6727d73154a4425e61db6f54931"}
Jul 27 18:29:16.227381 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:16.227331    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l6rrw" podStartSLOduration=2.803151191 podStartE2EDuration="16.227315426s" totalImagesPullingTime="13.424164235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:29:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:16.226794547 +0000 UTC m=+16.661052159" watchObservedRunningTime="2026-07-27 18:29:16.227315426 +0000 UTC m=+16.661573038"
Jul 27 18:29:16.243532 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:16.243492    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4g7qz" podStartSLOduration=2.785199977 podStartE2EDuration="16.243477605s" totalImagesPullingTime="13.458277628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:29:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:16.242878242 +0000 UTC m=+16.677135853" watchObservedRunningTime="2026-07-27 18:29:16.243477605 +0000 UTC m=+16.677735268"
Jul 27 18:29:16.259556 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:16.259521    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-cl7cw" podStartSLOduration=2.778819893 podStartE2EDuration="16.259507945s" totalImagesPullingTime="13.480688052s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:29:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:16.259087491 +0000 UTC m=+16.693345101" watchObservedRunningTime="2026-07-27 18:29:16.259507945 +0000 UTC m=+16.693765556"
Jul 27 18:29:16.778599 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:16.778556    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-original-pull-secret\") pod \"global-pull-secret-syncer-qrfwk\" (UID: \"662aa4f5-23c0-42d2-8b21-270c4b33e9a5\") " pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:16.778842 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:16.778625    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6751c8be-8944-44d2-8e25-f3fd6926225e-metrics-certs\") pod \"network-metrics-daemon-qc4qw\" (UID: \"6751c8be-8944-44d2-8e25-f3fd6926225e\") " pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:16.778842 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:16.778733    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 18:29:16.778842 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:16.778759    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 18:29:16.778842 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:16.778805    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-original-pull-secret podName:662aa4f5-23c0-42d2-8b21-270c4b33e9a5 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:32.778785701 +0000 UTC m=+33.213043302 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-original-pull-secret") pod "global-pull-secret-syncer-qrfwk" (UID: "662aa4f5-23c0-42d2-8b21-270c4b33e9a5") : object "kube-system"/"original-pull-secret" not registered
Jul 27 18:29:16.778842 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:16.778827    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6751c8be-8944-44d2-8e25-f3fd6926225e-metrics-certs podName:6751c8be-8944-44d2-8e25-f3fd6926225e nodeName:}" failed. No retries permitted until 2026-07-27 18:29:32.778816205 +0000 UTC m=+33.213073801 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6751c8be-8944-44d2-8e25-f3fd6926225e-metrics-certs") pod "network-metrics-daemon-qc4qw" (UID: "6751c8be-8944-44d2-8e25-f3fd6926225e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 18:29:16.879544 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:16.879500    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dftwx\" (UniqueName: \"kubernetes.io/projected/53874b39-22ec-4554-ab95-643a49ee8699-kube-api-access-dftwx\") pod \"network-check-target-jpq2j\" (UID: \"53874b39-22ec-4554-ab95-643a49ee8699\") " pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:16.879724 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:16.879653    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 18:29:16.879724 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:16.879675    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 18:29:16.879724 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:16.879686    2566 projected.go:196] Error preparing data for projected volume kube-api-access-dftwx for pod openshift-network-diagnostics/network-check-target-jpq2j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 18:29:16.879930 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:16.879748    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53874b39-22ec-4554-ab95-643a49ee8699-kube-api-access-dftwx podName:53874b39-22ec-4554-ab95-643a49ee8699 nodeName:}" failed. No retries permitted until 2026-07-27 18:29:32.879730655 +0000 UTC m=+33.313988259 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dftwx" (UniqueName: "kubernetes.io/projected/53874b39-22ec-4554-ab95-643a49ee8699-kube-api-access-dftwx") pod "network-check-target-jpq2j" (UID: "53874b39-22ec-4554-ab95-643a49ee8699") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 18:29:17.117496 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:17.117398    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:17.117496 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:17.117430    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:17.117703 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:17.117515    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:17.117703 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:17.117545    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:17.117949 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:17.117923    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:17.118135 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:17.118097    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:17.216107 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:17.216064    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f8mj4" event={"ID":"8c44c98c-c8de-4a6e-b827-3d56a456f1eb","Type":"ContainerStarted","Data":"6f77acefa23da263bf329e72bca1109dc779ded9955904fd97ae49f1d6d2b8c4"}
Jul 27 18:29:17.234250 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:17.234207    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-f8mj4" podStartSLOduration=3.862270121 podStartE2EDuration="17.234190656s" totalImagesPullingTime="13.371920535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:29:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:17.233683666 +0000 UTC m=+17.667941278" watchObservedRunningTime="2026-07-27 18:29:17.234190656 +0000 UTC m=+17.668448267"
Jul 27 18:29:18.221929 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:18.221887    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-196.ec2.internal" event={"ID":"b20c1cf26d726b4ecc2d495ed14b7a50","Type":"ContainerStarted","Data":"758f06f2f8e9114cf4d97dcaa8649e0052c71cac2d601cbea784e2012dfecbdb"}
Jul 27 18:29:18.243452 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:18.243401    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-196.ec2.internal" podStartSLOduration=17.243384102 podStartE2EDuration="17.243384102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:18.243056974 +0000 UTC m=+18.677314587" watchObservedRunningTime="2026-07-27 18:29:18.243384102 +0000 UTC m=+18.677641713"
Jul 27 18:29:18.560501 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:18.560474    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 18:29:19.019021 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:19.018617    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T18:29:18.560497666Z","UUID":"bc4b0015-0491-4895-9cb1-e71628bb9c04","Handler":null,"Name":"","Endpoint":""}
Jul 27 18:29:19.020368 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:19.020346    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 18:29:19.020502 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:19.020392    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 18:29:19.117804 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:19.117772    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:19.117977 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:19.117817    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:19.117977 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:19.117772    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:19.117977 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:19.117919    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:19.118176 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:19.118016    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:19.118176 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:19.118098    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:19.178332 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:19.178299    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l6rrw"
Jul 27 18:29:19.178854 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:19.178833    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l6rrw"
Jul 27 18:29:19.226229 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:19.226187    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv" event={"ID":"b81bd2e0-40bf-41ba-b0ab-2c86fdee9438","Type":"ContainerStarted","Data":"2950a0c79fd5c56e3b68900c59891990e750e3ab1b9abb190e4a6870228ba628"}
Jul 27 18:29:21.117924 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:21.117888    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:21.117924 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:21.117923    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:21.118435 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:21.117923    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:21.118435 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:21.118004    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:21.118435 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:21.118143    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:21.118435 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:21.118222    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:22.233940 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:22.233899    2566 generic.go:356] "Generic (PLEG): container finished" podID="18a80374-61a1-4cd0-806d-9578a7e70433" containerID="2d6dc2499c2c77cc4a932bb7ab48df924246d986ee918fe7a5f575ee9eebab27" exitCode=0
Jul 27 18:29:22.234421 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:22.233991    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qjwvr" event={"ID":"18a80374-61a1-4cd0-806d-9578a7e70433","Type":"ContainerDied","Data":"2d6dc2499c2c77cc4a932bb7ab48df924246d986ee918fe7a5f575ee9eebab27"}
Jul 27 18:29:22.236534 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:22.236507    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv" event={"ID":"b81bd2e0-40bf-41ba-b0ab-2c86fdee9438","Type":"ContainerStarted","Data":"75f2818bc66262d20c126930f4d656a3ece957206ddfd41a537e437f9ceaaf48"}
Jul 27 18:29:22.274402 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:22.274340    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bxjrv" podStartSLOduration=3.144159329 podStartE2EDuration="22.27431834s" totalImagesPullingTime="19.130159011s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:29:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:22.273948011 +0000 UTC m=+22.708205622" watchObservedRunningTime="2026-07-27 18:29:22.27431834 +0000 UTC m=+22.708575952"
Jul 27 18:29:23.117779 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:23.117740    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:23.117955 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:23.117740    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:23.117955 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:23.117880    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:23.117955 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:23.117741    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:23.118158 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:23.117954    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:23.118158 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:23.118082    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:24.242210 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:24.242176    2566 generic.go:356] "Generic (PLEG): container finished" podID="18a80374-61a1-4cd0-806d-9578a7e70433" containerID="8553aa2932bb35c09f9b3e56bd16a36b98228bf1bdf3357bd6571926de2cca21" exitCode=0
Jul 27 18:29:24.242629 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:24.242230    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qjwvr" event={"ID":"18a80374-61a1-4cd0-806d-9578a7e70433","Type":"ContainerDied","Data":"8553aa2932bb35c09f9b3e56bd16a36b98228bf1bdf3357bd6571926de2cca21"}
Jul 27 18:29:25.117429 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:25.117383    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:25.117429 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:25.117407    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:25.117649 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:25.117506    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:25.117649 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:25.117512    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:25.117649 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:25.117639    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:25.117781 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:25.117730    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:27.118392 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:27.118356    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:27.118392 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:27.118379    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:27.118879 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:27.118492    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:27.118879 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:27.118496    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:27.118879 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:27.118614    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:27.119021 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:27.118698    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:29.117870 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:29.117657    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:29.118411 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:29.117685    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:29.118411 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:29.117899    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:29.118411 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:29.117995    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:29.118411 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:29.117702    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:29.118411 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:29.118141    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:29.255940 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:29.255912    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 18:29:29.256287 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:29.256266    2566 generic.go:356] "Generic (PLEG): container finished" podID="fa213400-5979-496e-9d82-f6ba6a0f0333" containerID="6d162344d8cd17a13316cf96e64dd261da52aeac69d3b85dd7eb776b7ddb620c" exitCode=1
Jul 27 18:29:29.256371 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:29.256321    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t" event={"ID":"fa213400-5979-496e-9d82-f6ba6a0f0333","Type":"ContainerStarted","Data":"8d473533146340b9c33c76def42503d4660a6fc53fcedfd764a64590b098438d"}
Jul 27 18:29:29.256410 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:29.256375    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t" event={"ID":"fa213400-5979-496e-9d82-f6ba6a0f0333","Type":"ContainerStarted","Data":"dddb8451d84172506f2ff48b2ed6931065b7c11efd28d2ee140f488b838688bb"}
Jul 27 18:29:29.256410 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:29.256405    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t" event={"ID":"fa213400-5979-496e-9d82-f6ba6a0f0333","Type":"ContainerStarted","Data":"423f2ad35346ff7a3403173716eccf4dfbd4e29f10b5db2da09a1524ab331a52"}
Jul 27 18:29:29.256472 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:29.256434    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t" event={"ID":"fa213400-5979-496e-9d82-f6ba6a0f0333","Type":"ContainerStarted","Data":"7571af6e50f51b058c76d4fe031a0ce1de8a7c18bf4a4c03b401fc6b77d5864d"}
Jul 27 18:29:29.256472 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:29.256461    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t" event={"ID":"fa213400-5979-496e-9d82-f6ba6a0f0333","Type":"ContainerDied","Data":"6d162344d8cd17a13316cf96e64dd261da52aeac69d3b85dd7eb776b7ddb620c"}
Jul 27 18:29:29.256530 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:29.256492    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t" event={"ID":"fa213400-5979-496e-9d82-f6ba6a0f0333","Type":"ContainerStarted","Data":"6f17893431ea222fb815a5610583c8bd4b7bbe4bc63a32ba03385f83d8216c9c"}
Jul 27 18:29:30.260752 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:30.260719    2566 generic.go:356] "Generic (PLEG): container finished" podID="18a80374-61a1-4cd0-806d-9578a7e70433" containerID="2687440eedb1b8f377f493f08b470655281a4618c70a082f08a8e1f6d0508639" exitCode=0
Jul 27 18:29:30.261222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:30.260765    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qjwvr" event={"ID":"18a80374-61a1-4cd0-806d-9578a7e70433","Type":"ContainerDied","Data":"2687440eedb1b8f377f493f08b470655281a4618c70a082f08a8e1f6d0508639"}
Jul 27 18:29:31.118261 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:31.117832    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:31.118261 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:31.117936    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:31.118261 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:31.117941    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:31.118261 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:31.117979    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:31.118261 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:31.118094    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:31.118556 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:31.118255    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:31.145744 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:31.145707    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l6rrw"
Jul 27 18:29:31.145902 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:31.145841    2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 27 18:29:31.146392 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:31.146369    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l6rrw"
Jul 27 18:29:31.268348 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:31.268308    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 18:29:31.269173 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:31.269147    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t" event={"ID":"fa213400-5979-496e-9d82-f6ba6a0f0333","Type":"ContainerStarted","Data":"a670034b6e614673a5746b48a027e42da82661fe5a787d17cf34e34bb6b9d369"}
Jul 27 18:29:32.794138 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:32.794078    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-original-pull-secret\") pod \"global-pull-secret-syncer-qrfwk\" (UID: \"662aa4f5-23c0-42d2-8b21-270c4b33e9a5\") " pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:32.794550 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:32.794170    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6751c8be-8944-44d2-8e25-f3fd6926225e-metrics-certs\") pod \"network-metrics-daemon-qc4qw\" (UID: \"6751c8be-8944-44d2-8e25-f3fd6926225e\") " pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:32.794550 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:32.794239    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 18:29:32.794550 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:32.794311    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 18:29:32.794550 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:32.794318    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-original-pull-secret podName:662aa4f5-23c0-42d2-8b21-270c4b33e9a5 nodeName:}" failed. No retries permitted until 2026-07-27 18:30:04.794297282 +0000 UTC m=+65.228554875 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-original-pull-secret") pod "global-pull-secret-syncer-qrfwk" (UID: "662aa4f5-23c0-42d2-8b21-270c4b33e9a5") : object "kube-system"/"original-pull-secret" not registered
Jul 27 18:29:32.794550 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:32.794351    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6751c8be-8944-44d2-8e25-f3fd6926225e-metrics-certs podName:6751c8be-8944-44d2-8e25-f3fd6926225e nodeName:}" failed. No retries permitted until 2026-07-27 18:30:04.794340213 +0000 UTC m=+65.228597802 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6751c8be-8944-44d2-8e25-f3fd6926225e-metrics-certs") pod "network-metrics-daemon-qc4qw" (UID: "6751c8be-8944-44d2-8e25-f3fd6926225e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 18:29:32.894811 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:32.894767    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dftwx\" (UniqueName: \"kubernetes.io/projected/53874b39-22ec-4554-ab95-643a49ee8699-kube-api-access-dftwx\") pod \"network-check-target-jpq2j\" (UID: \"53874b39-22ec-4554-ab95-643a49ee8699\") " pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:32.894984 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:32.894942    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 18:29:32.894984 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:32.894967    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 18:29:32.894984 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:32.894977    2566 projected.go:196] Error preparing data for projected volume kube-api-access-dftwx for pod openshift-network-diagnostics/network-check-target-jpq2j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 18:29:32.895190 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:32.895030    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53874b39-22ec-4554-ab95-643a49ee8699-kube-api-access-dftwx podName:53874b39-22ec-4554-ab95-643a49ee8699 nodeName:}" failed. No retries permitted until 2026-07-27 18:30:04.895015186 +0000 UTC m=+65.329272775 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dftwx" (UniqueName: "kubernetes.io/projected/53874b39-22ec-4554-ab95-643a49ee8699-kube-api-access-dftwx") pod "network-check-target-jpq2j" (UID: "53874b39-22ec-4554-ab95-643a49ee8699") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 18:29:33.038163 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:33.038109    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cl7cw_4378195f-ed99-42aa-90e6-0fde9f5394f1/dns-node-resolver/0.log"
Jul 27 18:29:33.117744 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:33.117655    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:33.117744 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:33.117681    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:33.117942 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:33.117669    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:33.117942 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:33.117768    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:33.117942 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:33.117822    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:33.117942 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:33.117906    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:34.018589 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:34.018566    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4g7qz_0806ed23-3336-4327-ae3a-0da1e826f856/node-ca/0.log"
Jul 27 18:29:34.280160 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:34.279931    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 18:29:34.280604 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:34.280572    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t" event={"ID":"fa213400-5979-496e-9d82-f6ba6a0f0333","Type":"ContainerStarted","Data":"9283416770d9895751eded54f47dcc4a49a537d7f59922d7b43aee96bed4b459"}
Jul 27 18:29:34.280928 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:34.280902    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:34.281022 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:34.280993    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:34.281063 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:34.281048    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:34.281099 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:34.281077    2566 scope.go:117] "RemoveContainer" containerID="6d162344d8cd17a13316cf96e64dd261da52aeac69d3b85dd7eb776b7ddb620c"
Jul 27 18:29:34.288617 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:34.288057    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qjwvr" event={"ID":"18a80374-61a1-4cd0-806d-9578a7e70433","Type":"ContainerStarted","Data":"42b1ddb3f58db6e0f798dbd20334cd45f8eb241ade2ad05f47cba68bdb2e18c9"}
Jul 27 18:29:34.301001 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:34.300901    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:34.304847 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:34.304818    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:29:35.118134 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:35.118095    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:35.118718 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:35.118156    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:35.118718 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:35.118210    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:35.118718 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:35.118227    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:35.118718 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:35.118320    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:35.118718 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:35.118413    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:35.293245 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:35.293213    2566 generic.go:356] "Generic (PLEG): container finished" podID="18a80374-61a1-4cd0-806d-9578a7e70433" containerID="42b1ddb3f58db6e0f798dbd20334cd45f8eb241ade2ad05f47cba68bdb2e18c9" exitCode=0
Jul 27 18:29:35.293384 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:35.293295    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qjwvr" event={"ID":"18a80374-61a1-4cd0-806d-9578a7e70433","Type":"ContainerDied","Data":"42b1ddb3f58db6e0f798dbd20334cd45f8eb241ade2ad05f47cba68bdb2e18c9"}
Jul 27 18:29:35.297160 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:35.297141    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 18:29:35.297511 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:35.297489    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t" event={"ID":"fa213400-5979-496e-9d82-f6ba6a0f0333","Type":"ContainerStarted","Data":"70f4f50fbd05307fce2880aead076658100a0a6a26eedd3d001c247036560801"}
Jul 27 18:29:35.349071 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:35.349021    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t" podStartSLOduration=8.436606716 podStartE2EDuration="35.349002007s" totalImagesPullingTime="26.912395291s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:29:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=2 observedRunningTime="2026-07-27 18:29:35.348832862 +0000 UTC m=+35.783090473" watchObservedRunningTime="2026-07-27 18:29:35.349002007 +0000 UTC m=+35.783259619"
Jul 27 18:29:36.302881 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:36.302843    2566 generic.go:356] "Generic (PLEG): container finished" podID="18a80374-61a1-4cd0-806d-9578a7e70433" containerID="1eaf8e73a147a5d8af200a9df1a66c576eff21ba4bb9ece80b6052394e14a209" exitCode=0
Jul 27 18:29:36.303380 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:36.302888    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qjwvr" event={"ID":"18a80374-61a1-4cd0-806d-9578a7e70433","Type":"ContainerDied","Data":"1eaf8e73a147a5d8af200a9df1a66c576eff21ba4bb9ece80b6052394e14a209"}
Jul 27 18:29:37.117772 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:37.117740    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:37.117938 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:37.117836    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:37.117938 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:37.117750    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:37.117938 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:37.117879    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:37.118035 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:37.117943    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:37.118075 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:37.118051    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:37.308605 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:37.308575    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qjwvr" event={"ID":"18a80374-61a1-4cd0-806d-9578a7e70433","Type":"ContainerStarted","Data":"2617aeb71bcdc61a3fb95e911a93edb2921fc05d5e09dd03d2d110004a26201b"}
Jul 27 18:29:37.334792 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:37.334744    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qjwvr" podStartSLOduration=8.266676967 podStartE2EDuration="37.334728856s" totalImagesPullingTime="29.068051889s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:29:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:37.33417518 +0000 UTC m=+37.768432791" watchObservedRunningTime="2026-07-27 18:29:37.334728856 +0000 UTC m=+37.768986467"
Jul 27 18:29:39.117814 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:39.117774    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:39.117814 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:39.117801    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:39.117814 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:39.117801    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:39.118331 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:39.117885    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:39.118331 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:39.117985    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:39.118331 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:39.118070    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:41.118364 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:41.118332    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:41.118364 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:41.118350    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:41.118922 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:41.118440    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:41.118922 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:41.118454    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:41.118922 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:41.118521    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:41.118922 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:41.118604    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:43.118158 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:43.118104    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:43.118608 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:43.118104    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:43.118608 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:43.118241    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:43.118608 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:43.118104    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:43.118608 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:43.118304    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:43.118608 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:43.118383    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:45.118381 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:45.118348    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:45.118722 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:45.118349    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:45.118722 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:45.118453    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:45.118722 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:45.118349    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:45.118722 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:45.118526    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:45.118722 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:45.118607    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:47.118193 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:47.118148    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:47.118603 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:47.118205    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:47.118603 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:47.118290    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:47.118603 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:47.118294    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:47.118603 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:47.118352    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:47.118603 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:47.118409    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:49.118384 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:49.118350    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:49.118384 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:49.118374    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:49.118780 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:49.118469    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:49.118780 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:49.118543    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:49.118780 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:49.118637    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:49.118780 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:49.118743    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:51.118257 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:51.118221    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:51.118257 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:51.118255    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:51.118753 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:51.118221    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:51.118753 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:51.118332    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:51.118753 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:51.118400    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:51.118753 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:51.118534    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:53.117736 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:53.117704    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:53.118189 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:53.117704    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:53.118189 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:53.117804    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:53.118189 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:53.117704    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:53.118189 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:53.117876    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:53.118189 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:53.117937    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:55.118138 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:55.117940    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:55.118595 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:55.117944    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:55.118595 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:55.118254    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:55.118595 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:55.117960    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:55.118595 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:55.118305    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:55.118595 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:55.118378    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:57.117846 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:57.117812    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:57.117846 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:57.117843    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:57.118307 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:57.117843    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:57.118307 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:57.117920    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:57.118307 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:57.118012    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:57.118307 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:57.118083    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:57.417799 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:57.417720    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qc4qw"]
Jul 27 18:29:57.417947 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:57.417849    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:57.418024 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:57.417941    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:57.421045 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:57.421007    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qrfwk"]
Jul 27 18:29:57.421365 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:57.421348    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:57.421470 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:57.421345    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jpq2j"]
Jul 27 18:29:57.421531 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:57.421480    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:29:57.421586 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:57.421529    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:57.421638 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:57.421612    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:59.117556 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:59.117517    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:29:59.117556 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:59.117560    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:29:59.118256 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:29:59.117562    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:29:59.118256 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:59.117641    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:29:59.118256 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:59.117727    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:29:59.118256 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:29:59.117791    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:30:01.117705 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:01.117667    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:30:01.118231 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:01.117770    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:30:01.118231 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:30:01.117789    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpq2j" podUID="53874b39-22ec-4554-ab95-643a49ee8699"
Jul 27 18:30:01.118231 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:01.117801    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:30:01.118231 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:30:01.117876    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrfwk" podUID="662aa4f5-23c0-42d2-8b21-270c4b33e9a5"
Jul 27 18:30:01.118231 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:30:01.117953    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc4qw" podUID="6751c8be-8944-44d2-8e25-f3fd6926225e"
Jul 27 18:30:01.830271 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:01.830237    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-196.ec2.internal" event="NodeReady"
Jul 27 18:30:01.830456 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:01.830370    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 18:30:01.881073 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:01.881038    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sjj6g"]
Jul 27 18:30:01.902570 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:01.902544    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ft289"]
Jul 27 18:30:01.902737 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:01.902685    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sjj6g"
Jul 27 18:30:01.905415 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:01.905392    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xfn8t\""
Jul 27 18:30:01.905758 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:01.905740    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 18:30:01.905881 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:01.905798    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 18:30:01.905881 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:01.905835    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 18:30:01.926152 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:01.925974    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sjj6g"]
Jul 27 18:30:01.926256 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:01.926164    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ft289"]
Jul 27 18:30:01.926256 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:01.926082    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ft289"
Jul 27 18:30:01.932063 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:01.932039    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-584h7\""
Jul 27 18:30:01.932167 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:01.932146    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 18:30:01.932736 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:01.932710    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 18:30:01.932736 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:01.932714    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 18:30:01.932884 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:01.932715    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 18:30:01.982021 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:01.981941    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-r7j9k"]
Jul 27 18:30:02.004224 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.004192    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r7j9k"]
Jul 27 18:30:02.004380 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.004326    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r7j9k"
Jul 27 18:30:02.007222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.007190    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9pm47\""
Jul 27 18:30:02.007222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.007213    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 18:30:02.007409 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.007292    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 18:30:02.009908 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.009882    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db0becf5-bc5c-4511-8ccd-43c17b98293e-cert\") pod \"ingress-canary-sjj6g\" (UID: \"db0becf5-bc5c-4511-8ccd-43c17b98293e\") " pod="openshift-ingress-canary/ingress-canary-sjj6g"
Jul 27 18:30:02.010044 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.009925    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d94bb424-b152-4636-9e19-ba2a0a55ceb4-crio-socket\") pod \"insights-runtime-extractor-ft289\" (UID: \"d94bb424-b152-4636-9e19-ba2a0a55ceb4\") " pod="openshift-insights/insights-runtime-extractor-ft289"
Jul 27 18:30:02.010044 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.009952    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d94bb424-b152-4636-9e19-ba2a0a55ceb4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ft289\" (UID: \"d94bb424-b152-4636-9e19-ba2a0a55ceb4\") " pod="openshift-insights/insights-runtime-extractor-ft289"
Jul 27 18:30:02.010044 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.010020    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d94bb424-b152-4636-9e19-ba2a0a55ceb4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ft289\" (UID: \"d94bb424-b152-4636-9e19-ba2a0a55ceb4\") " pod="openshift-insights/insights-runtime-extractor-ft289"
Jul 27 18:30:02.010211 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.010047    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87944\" (UniqueName: \"kubernetes.io/projected/d94bb424-b152-4636-9e19-ba2a0a55ceb4-kube-api-access-87944\") pod \"insights-runtime-extractor-ft289\" (UID: \"d94bb424-b152-4636-9e19-ba2a0a55ceb4\") " pod="openshift-insights/insights-runtime-extractor-ft289"
Jul 27 18:30:02.010211 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.010166    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbvqj\" (UniqueName: \"kubernetes.io/projected/db0becf5-bc5c-4511-8ccd-43c17b98293e-kube-api-access-mbvqj\") pod \"ingress-canary-sjj6g\" (UID: \"db0becf5-bc5c-4511-8ccd-43c17b98293e\") " pod="openshift-ingress-canary/ingress-canary-sjj6g"
Jul 27 18:30:02.010211 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.010188    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d94bb424-b152-4636-9e19-ba2a0a55ceb4-data-volume\") pod \"insights-runtime-extractor-ft289\" (UID: \"d94bb424-b152-4636-9e19-ba2a0a55ceb4\") " pod="openshift-insights/insights-runtime-extractor-ft289"
Jul 27 18:30:02.091062 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.091028    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hfg4n"]
Jul 27 18:30:02.111080 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.111048    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbvqj\" (UniqueName: \"kubernetes.io/projected/db0becf5-bc5c-4511-8ccd-43c17b98293e-kube-api-access-mbvqj\") pod \"ingress-canary-sjj6g\" (UID: \"db0becf5-bc5c-4511-8ccd-43c17b98293e\") " pod="openshift-ingress-canary/ingress-canary-sjj6g"
Jul 27 18:30:02.111256 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.111095    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/423413aa-a12b-40ce-9da1-02de296eb175-config-volume\") pod \"dns-default-r7j9k\" (UID: \"423413aa-a12b-40ce-9da1-02de296eb175\") " pod="openshift-dns/dns-default-r7j9k"
Jul 27 18:30:02.111256 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.111142    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d94bb424-b152-4636-9e19-ba2a0a55ceb4-data-volume\") pod \"insights-runtime-extractor-ft289\" (UID: \"d94bb424-b152-4636-9e19-ba2a0a55ceb4\") " pod="openshift-insights/insights-runtime-extractor-ft289"
Jul 27 18:30:02.111370 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.111312    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9klvj\" (UniqueName: \"kubernetes.io/projected/423413aa-a12b-40ce-9da1-02de296eb175-kube-api-access-9klvj\") pod \"dns-default-r7j9k\" (UID: \"423413aa-a12b-40ce-9da1-02de296eb175\") " pod="openshift-dns/dns-default-r7j9k"
Jul 27 18:30:02.111429 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.111378    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db0becf5-bc5c-4511-8ccd-43c17b98293e-cert\") pod \"ingress-canary-sjj6g\" (UID: \"db0becf5-bc5c-4511-8ccd-43c17b98293e\") " pod="openshift-ingress-canary/ingress-canary-sjj6g"
Jul 27 18:30:02.111429 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.111406    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d94bb424-b152-4636-9e19-ba2a0a55ceb4-crio-socket\") pod \"insights-runtime-extractor-ft289\" (UID: \"d94bb424-b152-4636-9e19-ba2a0a55ceb4\") " pod="openshift-insights/insights-runtime-extractor-ft289"
Jul 27 18:30:02.111526 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.111431    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d94bb424-b152-4636-9e19-ba2a0a55ceb4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ft289\" (UID: \"d94bb424-b152-4636-9e19-ba2a0a55ceb4\") " pod="openshift-insights/insights-runtime-extractor-ft289"
Jul 27 18:30:02.111526 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.111482    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/423413aa-a12b-40ce-9da1-02de296eb175-tmp-dir\") pod \"dns-default-r7j9k\" (UID: \"423413aa-a12b-40ce-9da1-02de296eb175\") " pod="openshift-dns/dns-default-r7j9k"
Jul 27 18:30:02.111526 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.111506    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d94bb424-b152-4636-9e19-ba2a0a55ceb4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ft289\" (UID: \"d94bb424-b152-4636-9e19-ba2a0a55ceb4\") " pod="openshift-insights/insights-runtime-extractor-ft289"
Jul 27 18:30:02.111672 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.111531    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87944\" (UniqueName: \"kubernetes.io/projected/d94bb424-b152-4636-9e19-ba2a0a55ceb4-kube-api-access-87944\") pod \"insights-runtime-extractor-ft289\" (UID: \"d94bb424-b152-4636-9e19-ba2a0a55ceb4\") " pod="openshift-insights/insights-runtime-extractor-ft289"
Jul 27 18:30:02.111672 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.111534    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d94bb424-b152-4636-9e19-ba2a0a55ceb4-data-volume\") pod \"insights-runtime-extractor-ft289\" (UID: \"d94bb424-b152-4636-9e19-ba2a0a55ceb4\") " pod="openshift-insights/insights-runtime-extractor-ft289"
Jul 27 18:30:02.111672 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.111557    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d94bb424-b152-4636-9e19-ba2a0a55ceb4-crio-socket\") pod \"insights-runtime-extractor-ft289\" (UID: \"d94bb424-b152-4636-9e19-ba2a0a55ceb4\") " pod="openshift-insights/insights-runtime-extractor-ft289"
Jul 27 18:30:02.111672 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.111562    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/423413aa-a12b-40ce-9da1-02de296eb175-metrics-tls\") pod \"dns-default-r7j9k\" (UID: \"423413aa-a12b-40ce-9da1-02de296eb175\") " pod="openshift-dns/dns-default-r7j9k"
Jul 27 18:30:02.112222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.112199    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d94bb424-b152-4636-9e19-ba2a0a55ceb4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ft289\" (UID: \"d94bb424-b152-4636-9e19-ba2a0a55ceb4\") " pod="openshift-insights/insights-runtime-extractor-ft289"
Jul 27 18:30:02.114317 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.114294    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hfg4n"]
Jul 27 18:30:02.114407 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.114362    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hfg4n"
Jul 27 18:30:02.115488 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.115467    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d94bb424-b152-4636-9e19-ba2a0a55ceb4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ft289\" (UID: \"d94bb424-b152-4636-9e19-ba2a0a55ceb4\") " pod="openshift-insights/insights-runtime-extractor-ft289"
Jul 27 18:30:02.115650 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.115635    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db0becf5-bc5c-4511-8ccd-43c17b98293e-cert\") pod \"ingress-canary-sjj6g\" (UID: \"db0becf5-bc5c-4511-8ccd-43c17b98293e\") " pod="openshift-ingress-canary/ingress-canary-sjj6g"
Jul 27 18:30:02.117601 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.117583    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-wlg2s\""
Jul 27 18:30:02.117874 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.117862    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 27 18:30:02.134339 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.134317    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87944\" (UniqueName: \"kubernetes.io/projected/d94bb424-b152-4636-9e19-ba2a0a55ceb4-kube-api-access-87944\") pod \"insights-runtime-extractor-ft289\" (UID: \"d94bb424-b152-4636-9e19-ba2a0a55ceb4\") " pod="openshift-insights/insights-runtime-extractor-ft289"
Jul 27 18:30:02.135041 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.135019    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbvqj\" (UniqueName: \"kubernetes.io/projected/db0becf5-bc5c-4511-8ccd-43c17b98293e-kube-api-access-mbvqj\") pod \"ingress-canary-sjj6g\" (UID: \"db0becf5-bc5c-4511-8ccd-43c17b98293e\") " pod="openshift-ingress-canary/ingress-canary-sjj6g"
Jul 27 18:30:02.212154 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.212091    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sjj6g"
Jul 27 18:30:02.215749 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.213311    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/423413aa-a12b-40ce-9da1-02de296eb175-tmp-dir\") pod \"dns-default-r7j9k\" (UID: \"423413aa-a12b-40ce-9da1-02de296eb175\") " pod="openshift-dns/dns-default-r7j9k"
Jul 27 18:30:02.215749 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.213364    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/423413aa-a12b-40ce-9da1-02de296eb175-metrics-tls\") pod \"dns-default-r7j9k\" (UID: \"423413aa-a12b-40ce-9da1-02de296eb175\") " pod="openshift-dns/dns-default-r7j9k"
Jul 27 18:30:02.215749 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.213401    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8c0ca6e3-528f-4cd5-841c-4ab0ccedf99f-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-hfg4n\" (UID: \"8c0ca6e3-528f-4cd5-841c-4ab0ccedf99f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hfg4n"
Jul 27 18:30:02.215749 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.213465    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/423413aa-a12b-40ce-9da1-02de296eb175-config-volume\") pod \"dns-default-r7j9k\" (UID: \"423413aa-a12b-40ce-9da1-02de296eb175\") " pod="openshift-dns/dns-default-r7j9k"
Jul 27 18:30:02.215749 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.213497    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9klvj\" (UniqueName: \"kubernetes.io/projected/423413aa-a12b-40ce-9da1-02de296eb175-kube-api-access-9klvj\") pod \"dns-default-r7j9k\" (UID: \"423413aa-a12b-40ce-9da1-02de296eb175\") " pod="openshift-dns/dns-default-r7j9k"
Jul 27 18:30:02.215749 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.215753    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/423413aa-a12b-40ce-9da1-02de296eb175-tmp-dir\") pod \"dns-default-r7j9k\" (UID: \"423413aa-a12b-40ce-9da1-02de296eb175\") " pod="openshift-dns/dns-default-r7j9k"
Jul 27 18:30:02.216621 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.216596    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/423413aa-a12b-40ce-9da1-02de296eb175-config-volume\") pod \"dns-default-r7j9k\" (UID: \"423413aa-a12b-40ce-9da1-02de296eb175\") " pod="openshift-dns/dns-default-r7j9k"
Jul 27 18:30:02.218665 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.218642    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/423413aa-a12b-40ce-9da1-02de296eb175-metrics-tls\") pod \"dns-default-r7j9k\" (UID: \"423413aa-a12b-40ce-9da1-02de296eb175\") " pod="openshift-dns/dns-default-r7j9k"
Jul 27 18:30:02.224264 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.224246    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9klvj\" (UniqueName: \"kubernetes.io/projected/423413aa-a12b-40ce-9da1-02de296eb175-kube-api-access-9klvj\") pod \"dns-default-r7j9k\" (UID: \"423413aa-a12b-40ce-9da1-02de296eb175\") " pod="openshift-dns/dns-default-r7j9k"
Jul 27 18:30:02.233879 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.233837    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ft289"
Jul 27 18:30:02.314525 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.314482    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r7j9k"
Jul 27 18:30:02.314690 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.314595    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8c0ca6e3-528f-4cd5-841c-4ab0ccedf99f-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-hfg4n\" (UID: \"8c0ca6e3-528f-4cd5-841c-4ab0ccedf99f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hfg4n"
Jul 27 18:30:02.317328 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.317302    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8c0ca6e3-528f-4cd5-841c-4ab0ccedf99f-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-hfg4n\" (UID: \"8c0ca6e3-528f-4cd5-841c-4ab0ccedf99f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hfg4n"
Jul 27 18:30:02.420039 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.417933    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-b2b27"]
Jul 27 18:30:02.431094 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.430842    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hfg4n"
Jul 27 18:30:02.431604 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.431583    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-b2b27"
Jul 27 18:30:02.442342 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.442322    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-4psrr\""
Jul 27 18:30:02.442553 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.442537    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 27 18:30:02.442707 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.442692    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 27 18:30:02.479284 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.479242    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-b2b27"]
Jul 27 18:30:02.495041 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.494963    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sjj6g"]
Jul 27 18:30:02.497762 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:30:02.497735    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0becf5_bc5c_4511_8ccd_43c17b98293e.slice/crio-0557b4c914c8f85194d5cb58301f71ee53a1b0bf96281200ee790c1236f91c3a WatchSource:0}: Error finding container 0557b4c914c8f85194d5cb58301f71ee53a1b0bf96281200ee790c1236f91c3a: Status 404 returned error can't find the container with id 0557b4c914c8f85194d5cb58301f71ee53a1b0bf96281200ee790c1236f91c3a
Jul 27 18:30:02.503575 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.503399    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ft289"]
Jul 27 18:30:02.506776 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:30:02.506747    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd94bb424_b152_4636_9e19_ba2a0a55ceb4.slice/crio-1f4e2f39446502d9c27c65ac2e2acaca01ad3cfb460bb812439be536aff1b887 WatchSource:0}: Error finding container 1f4e2f39446502d9c27c65ac2e2acaca01ad3cfb460bb812439be536aff1b887: Status 404 returned error can't find the container with id 1f4e2f39446502d9c27c65ac2e2acaca01ad3cfb460bb812439be536aff1b887
Jul 27 18:30:02.515979 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.515950    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r7j9k"]
Jul 27 18:30:02.516100 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.516077    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62v47\" (UniqueName: \"kubernetes.io/projected/0cc5553b-6803-4a64-a3b4-a4ef998a14dc-kube-api-access-62v47\") pod \"downloads-7c97dc4555-b2b27\" (UID: \"0cc5553b-6803-4a64-a3b4-a4ef998a14dc\") " pod="openshift-console/downloads-7c97dc4555-b2b27"
Jul 27 18:30:02.518920 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:30:02.518899    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423413aa_a12b_40ce_9da1_02de296eb175.slice/crio-922c4b51cabf4564b971ac4a60eee07d8871a21f6209be34d72b9fbde9c05444 WatchSource:0}: Error finding container 922c4b51cabf4564b971ac4a60eee07d8871a21f6209be34d72b9fbde9c05444: Status 404 returned error can't find the container with id 922c4b51cabf4564b971ac4a60eee07d8871a21f6209be34d72b9fbde9c05444
Jul 27 18:30:02.576693 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.576660    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hfg4n"]
Jul 27 18:30:02.582153 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:30:02.580757    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c0ca6e3_528f_4cd5_841c_4ab0ccedf99f.slice/crio-7abdc203b221be6e087f25669e16ce21ffc31b1daa98e96441dfb989f4e32e61 WatchSource:0}: Error finding container 7abdc203b221be6e087f25669e16ce21ffc31b1daa98e96441dfb989f4e32e61: Status 404 returned error can't find the container with id 7abdc203b221be6e087f25669e16ce21ffc31b1daa98e96441dfb989f4e32e61
Jul 27 18:30:02.617204 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.617174    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62v47\" (UniqueName: \"kubernetes.io/projected/0cc5553b-6803-4a64-a3b4-a4ef998a14dc-kube-api-access-62v47\") pod \"downloads-7c97dc4555-b2b27\" (UID: \"0cc5553b-6803-4a64-a3b4-a4ef998a14dc\") " pod="openshift-console/downloads-7c97dc4555-b2b27"
Jul 27 18:30:02.627096 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.627069    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62v47\" (UniqueName: \"kubernetes.io/projected/0cc5553b-6803-4a64-a3b4-a4ef998a14dc-kube-api-access-62v47\") pod \"downloads-7c97dc4555-b2b27\" (UID: \"0cc5553b-6803-4a64-a3b4-a4ef998a14dc\") " pod="openshift-console/downloads-7c97dc4555-b2b27"
Jul 27 18:30:02.743183 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.743077    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-b2b27"
Jul 27 18:30:02.901018 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:02.900988    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-b2b27"]
Jul 27 18:30:02.904370 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:30:02.904342    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc5553b_6803_4a64_a3b4_a4ef998a14dc.slice/crio-f690b687eda9df878e086bc7012cdf768aa17fe2c450f9ffd70879ac8f002703 WatchSource:0}: Error finding container f690b687eda9df878e086bc7012cdf768aa17fe2c450f9ffd70879ac8f002703: Status 404 returned error can't find the container with id f690b687eda9df878e086bc7012cdf768aa17fe2c450f9ffd70879ac8f002703
Jul 27 18:30:03.118413 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:03.118380    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:30:03.118819 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:03.118380    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:30:03.118819 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:03.118381    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:30:03.121582 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:03.121568    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fn8tb\""
Jul 27 18:30:03.121631 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:03.121582    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 18:30:03.122852 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:03.122839    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 18:30:03.123062 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:03.123030    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 18:30:03.123062 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:03.123033    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 18:30:03.123234 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:03.123063    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bwbqc\""
Jul 27 18:30:03.372107 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:03.371825    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sjj6g" event={"ID":"db0becf5-bc5c-4511-8ccd-43c17b98293e","Type":"ContainerStarted","Data":"0557b4c914c8f85194d5cb58301f71ee53a1b0bf96281200ee790c1236f91c3a"}
Jul 27 18:30:03.373038 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:03.373010    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-b2b27" event={"ID":"0cc5553b-6803-4a64-a3b4-a4ef998a14dc","Type":"ContainerStarted","Data":"f690b687eda9df878e086bc7012cdf768aa17fe2c450f9ffd70879ac8f002703"}
Jul 27 18:30:03.374230 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:03.374210    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hfg4n" event={"ID":"8c0ca6e3-528f-4cd5-841c-4ab0ccedf99f","Type":"ContainerStarted","Data":"7abdc203b221be6e087f25669e16ce21ffc31b1daa98e96441dfb989f4e32e61"}
Jul 27 18:30:03.375263 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:03.375240    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r7j9k" event={"ID":"423413aa-a12b-40ce-9da1-02de296eb175","Type":"ContainerStarted","Data":"922c4b51cabf4564b971ac4a60eee07d8871a21f6209be34d72b9fbde9c05444"}
Jul 27 18:30:03.376491 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:03.376469    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ft289" event={"ID":"d94bb424-b152-4636-9e19-ba2a0a55ceb4","Type":"ContainerStarted","Data":"1f4e2f39446502d9c27c65ac2e2acaca01ad3cfb460bb812439be536aff1b887"}
Jul 27 18:30:04.837449 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:04.837408    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-original-pull-secret\") pod \"global-pull-secret-syncer-qrfwk\" (UID: \"662aa4f5-23c0-42d2-8b21-270c4b33e9a5\") " pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:30:04.837984 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:04.837482    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6751c8be-8944-44d2-8e25-f3fd6926225e-metrics-certs\") pod \"network-metrics-daemon-qc4qw\" (UID: \"6751c8be-8944-44d2-8e25-f3fd6926225e\") " pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:30:04.852700 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:04.852639    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/662aa4f5-23c0-42d2-8b21-270c4b33e9a5-original-pull-secret\") pod \"global-pull-secret-syncer-qrfwk\" (UID: \"662aa4f5-23c0-42d2-8b21-270c4b33e9a5\") " pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:30:04.859878 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:04.859612    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6751c8be-8944-44d2-8e25-f3fd6926225e-metrics-certs\") pod \"network-metrics-daemon-qc4qw\" (UID: \"6751c8be-8944-44d2-8e25-f3fd6926225e\") " pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:30:04.938660 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:04.938158    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dftwx\" (UniqueName: \"kubernetes.io/projected/53874b39-22ec-4554-ab95-643a49ee8699-kube-api-access-dftwx\") pod \"network-check-target-jpq2j\" (UID: \"53874b39-22ec-4554-ab95-643a49ee8699\") " pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:30:04.946930 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:04.946688    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrfwk"
Jul 27 18:30:04.947831 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:04.947745    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dftwx\" (UniqueName: \"kubernetes.io/projected/53874b39-22ec-4554-ab95-643a49ee8699-kube-api-access-dftwx\") pod \"network-check-target-jpq2j\" (UID: \"53874b39-22ec-4554-ab95-643a49ee8699\") " pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:30:04.950853 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:04.950831    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc4qw"
Jul 27 18:30:05.240711 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:05.240626    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:30:06.317033 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:06.316410    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tdb9t"
Jul 27 18:30:10.123777 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:10.123749    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qrfwk"]
Jul 27 18:30:10.398230 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:10.398193    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qrfwk" event={"ID":"662aa4f5-23c0-42d2-8b21-270c4b33e9a5","Type":"ContainerStarted","Data":"626cc17e95896960910cfe0988f8e8c58f74c6100ac1b79744ae90c1e964c2f4"}
Jul 27 18:30:10.494878 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:10.494848    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qc4qw"]
Jul 27 18:30:10.774810 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:30:10.774777    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6751c8be_8944_44d2_8e25_f3fd6926225e.slice/crio-25d85a68c64ddad809bcf68b98eb04628560357d5c66f33d0c3ca77e78ec8dea WatchSource:0}: Error finding container 25d85a68c64ddad809bcf68b98eb04628560357d5c66f33d0c3ca77e78ec8dea: Status 404 returned error can't find the container with id 25d85a68c64ddad809bcf68b98eb04628560357d5c66f33d0c3ca77e78ec8dea
Jul 27 18:30:10.937160 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:10.936038    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jpq2j"]
Jul 27 18:30:10.946030 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:30:10.945979    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53874b39_22ec_4554_ab95_643a49ee8699.slice/crio-3cc1128db92c92bdbd34600cf5e03292fa1713a24aa159fbf3f9693a031f5594 WatchSource:0}: Error finding container 3cc1128db92c92bdbd34600cf5e03292fa1713a24aa159fbf3f9693a031f5594: Status 404 returned error can't find the container with id 3cc1128db92c92bdbd34600cf5e03292fa1713a24aa159fbf3f9693a031f5594
Jul 27 18:30:11.293106 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.292040    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85d76996c4-2r2cw"]
Jul 27 18:30:11.295224 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.294335    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85d76996c4-2r2cw"
Jul 27 18:30:11.297470 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.297450    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-96s4l\""
Jul 27 18:30:11.299826 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.299805    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 18:30:11.299926 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.299893    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 18:30:11.300081 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.300066    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 18:30:11.300170 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.300160    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 18:30:11.300380 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.300362    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 18:30:11.309683 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.309661    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85d76996c4-2r2cw"]
Jul 27 18:30:11.384266 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.384016    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff718661-da68-46b2-afa6-3ef3927c722c-console-config\") pod \"console-85d76996c4-2r2cw\" (UID: \"ff718661-da68-46b2-afa6-3ef3927c722c\") " pod="openshift-console/console-85d76996c4-2r2cw"
Jul 27 18:30:11.384266 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.384058    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff718661-da68-46b2-afa6-3ef3927c722c-service-ca\") pod \"console-85d76996c4-2r2cw\" (UID: \"ff718661-da68-46b2-afa6-3ef3927c722c\") " pod="openshift-console/console-85d76996c4-2r2cw"
Jul 27 18:30:11.384266 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.384086    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff718661-da68-46b2-afa6-3ef3927c722c-console-serving-cert\") pod \"console-85d76996c4-2r2cw\" (UID: \"ff718661-da68-46b2-afa6-3ef3927c722c\") " pod="openshift-console/console-85d76996c4-2r2cw"
Jul 27 18:30:11.384266 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.384101    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp8sm\" (UniqueName: \"kubernetes.io/projected/ff718661-da68-46b2-afa6-3ef3927c722c-kube-api-access-pp8sm\") pod \"console-85d76996c4-2r2cw\" (UID: \"ff718661-da68-46b2-afa6-3ef3927c722c\") " pod="openshift-console/console-85d76996c4-2r2cw"
Jul 27 18:30:11.384266 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.384161    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff718661-da68-46b2-afa6-3ef3927c722c-console-oauth-config\") pod \"console-85d76996c4-2r2cw\" (UID: \"ff718661-da68-46b2-afa6-3ef3927c722c\") " pod="openshift-console/console-85d76996c4-2r2cw"
Jul 27 18:30:11.384266 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.384177    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff718661-da68-46b2-afa6-3ef3927c722c-oauth-serving-cert\") pod \"console-85d76996c4-2r2cw\" (UID: \"ff718661-da68-46b2-afa6-3ef3927c722c\") " pod="openshift-console/console-85d76996c4-2r2cw"
Jul 27 18:30:11.405413 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.405335    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jpq2j" event={"ID":"53874b39-22ec-4554-ab95-643a49ee8699","Type":"ContainerStarted","Data":"3cc1128db92c92bdbd34600cf5e03292fa1713a24aa159fbf3f9693a031f5594"}
Jul 27 18:30:11.407769 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.407051    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hfg4n" event={"ID":"8c0ca6e3-528f-4cd5-841c-4ab0ccedf99f","Type":"ContainerStarted","Data":"2ea948cc246ae85ae9f8a1280d090c98abed400a2fa110c28503ce31933bb762"}
Jul 27 18:30:11.407769 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.407405    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hfg4n"
Jul 27 18:30:11.414887 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.413978    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r7j9k" event={"ID":"423413aa-a12b-40ce-9da1-02de296eb175","Type":"ContainerStarted","Data":"5232816014a613f522e047ae4fdf1bea70b1d9500da0c574d87da700d99dec48"}
Jul 27 18:30:11.414887 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.414038    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r7j9k" event={"ID":"423413aa-a12b-40ce-9da1-02de296eb175","Type":"ContainerStarted","Data":"2f897b0d86aca542e5787814ba5d97863eedee5a0387a017a2ca7394b08ecc4a"}
Jul 27 18:30:11.414887 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.414578    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-r7j9k"
Jul 27 18:30:11.414887 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.414823    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hfg4n"
Jul 27 18:30:11.417952 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.417870    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ft289" event={"ID":"d94bb424-b152-4636-9e19-ba2a0a55ceb4","Type":"ContainerStarted","Data":"4f93656b629b95371d2a1a2103961b5a396ceee438db37d92fe36a3cecc772ac"}
Jul 27 18:30:11.420561 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.420110    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sjj6g" event={"ID":"db0becf5-bc5c-4511-8ccd-43c17b98293e","Type":"ContainerStarted","Data":"4fdbf337ab56f440490d7a87c90dee838a5a6f905fc1b0dcd80dfd61ad7cf4d9"}
Jul 27 18:30:11.421936 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.421911    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qc4qw" event={"ID":"6751c8be-8944-44d2-8e25-f3fd6926225e","Type":"ContainerStarted","Data":"25d85a68c64ddad809bcf68b98eb04628560357d5c66f33d0c3ca77e78ec8dea"}
Jul 27 18:30:11.427000 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.426948    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hfg4n" podStartSLOduration=2.038998771 podStartE2EDuration="9.426930663s" totalImagesPullingTime="7.387931892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:30:11.426012197 +0000 UTC m=+71.860269801" watchObservedRunningTime="2026-07-27 18:30:11.426930663 +0000 UTC m=+71.861188277"
Jul 27 18:30:11.462797 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.462713    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-r7j9k" podStartSLOduration=2.2112671329999998 podStartE2EDuration="10.462690086s" totalImagesPullingTime="8.251422953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:30:11.444434594 +0000 UTC m=+71.878692205" watchObservedRunningTime="2026-07-27 18:30:11.462690086 +0000 UTC m=+71.896947702"
Jul 27 18:30:11.482348 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.482289    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sjj6g" podStartSLOduration=2.21318633 podStartE2EDuration="10.48226528s" totalImagesPullingTime="8.26907895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:30:11.46357382 +0000 UTC m=+71.897831432" watchObservedRunningTime="2026-07-27 18:30:11.48226528 +0000 UTC m=+71.916522891"
Jul 27 18:30:11.485054 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.485023    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff718661-da68-46b2-afa6-3ef3927c722c-console-serving-cert\") pod \"console-85d76996c4-2r2cw\" (UID: \"ff718661-da68-46b2-afa6-3ef3927c722c\") " pod="openshift-console/console-85d76996c4-2r2cw"
Jul 27 18:30:11.485200 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.485066    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp8sm\" (UniqueName: \"kubernetes.io/projected/ff718661-da68-46b2-afa6-3ef3927c722c-kube-api-access-pp8sm\") pod \"console-85d76996c4-2r2cw\" (UID: \"ff718661-da68-46b2-afa6-3ef3927c722c\") " pod="openshift-console/console-85d76996c4-2r2cw"
Jul 27 18:30:11.485200 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.485180    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff718661-da68-46b2-afa6-3ef3927c722c-console-oauth-config\") pod \"console-85d76996c4-2r2cw\" (UID: \"ff718661-da68-46b2-afa6-3ef3927c722c\") " pod="openshift-console/console-85d76996c4-2r2cw"
Jul 27 18:30:11.485319 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.485206    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff718661-da68-46b2-afa6-3ef3927c722c-oauth-serving-cert\") pod \"console-85d76996c4-2r2cw\" (UID: \"ff718661-da68-46b2-afa6-3ef3927c722c\") " pod="openshift-console/console-85d76996c4-2r2cw"
Jul 27 18:30:11.485319 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.485272    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff718661-da68-46b2-afa6-3ef3927c722c-console-config\") pod \"console-85d76996c4-2r2cw\" (UID: \"ff718661-da68-46b2-afa6-3ef3927c722c\") " pod="openshift-console/console-85d76996c4-2r2cw"
Jul 27 18:30:11.485319 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.485299    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff718661-da68-46b2-afa6-3ef3927c722c-service-ca\") pod \"console-85d76996c4-2r2cw\" (UID: \"ff718661-da68-46b2-afa6-3ef3927c722c\") " pod="openshift-console/console-85d76996c4-2r2cw"
Jul 27 18:30:11.486062 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.486031    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff718661-da68-46b2-afa6-3ef3927c722c-service-ca\") pod \"console-85d76996c4-2r2cw\" (UID: \"ff718661-da68-46b2-afa6-3ef3927c722c\") " pod="openshift-console/console-85d76996c4-2r2cw"
Jul 27 18:30:11.488405 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.488382    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff718661-da68-46b2-afa6-3ef3927c722c-console-config\") pod \"console-85d76996c4-2r2cw\" (UID: \"ff718661-da68-46b2-afa6-3ef3927c722c\") " pod="openshift-console/console-85d76996c4-2r2cw"
Jul 27 18:30:11.488940 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.488916    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff718661-da68-46b2-afa6-3ef3927c722c-oauth-serving-cert\") pod \"console-85d76996c4-2r2cw\" (UID: \"ff718661-da68-46b2-afa6-3ef3927c722c\") " pod="openshift-console/console-85d76996c4-2r2cw"
Jul 27 18:30:11.503879 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.503850    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp8sm\" (UniqueName: \"kubernetes.io/projected/ff718661-da68-46b2-afa6-3ef3927c722c-kube-api-access-pp8sm\") pod \"console-85d76996c4-2r2cw\" (UID: \"ff718661-da68-46b2-afa6-3ef3927c722c\") " pod="openshift-console/console-85d76996c4-2r2cw"
Jul 27 18:30:11.507054 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.507031    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff718661-da68-46b2-afa6-3ef3927c722c-console-oauth-config\") pod \"console-85d76996c4-2r2cw\" (UID: \"ff718661-da68-46b2-afa6-3ef3927c722c\") " pod="openshift-console/console-85d76996c4-2r2cw"
Jul 27 18:30:11.510228 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.510204    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff718661-da68-46b2-afa6-3ef3927c722c-console-serving-cert\") pod \"console-85d76996c4-2r2cw\" (UID: \"ff718661-da68-46b2-afa6-3ef3927c722c\") " pod="openshift-console/console-85d76996c4-2r2cw"
Jul 27 18:30:11.611660 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:11.611355    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85d76996c4-2r2cw"
Jul 27 18:30:12.126396 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:12.126365    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-shhvl"]
Jul 27 18:30:12.129050 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:12.129029    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-shhvl"
Jul 27 18:30:12.136720 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:12.134724    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 18:30:12.136720 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:12.134786    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 18:30:12.136720 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:12.134724    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 18:30:12.136720 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:12.135037    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 27 18:30:12.136720 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:12.135153    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 27 18:30:12.136720 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:12.135267    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9nkzm\""
Jul 27 18:30:12.143482 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:12.141438    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-shhvl"]
Jul 27 18:30:12.192931 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:12.191730    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-shhvl\" (UID: \"2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-shhvl"
Jul 27 18:30:12.192931 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:12.191783    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-shhvl\" (UID: \"2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-shhvl"
Jul 27 18:30:12.192931 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:12.191825    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn5cx\" (UniqueName: \"kubernetes.io/projected/2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50-kube-api-access-fn5cx\") pod \"prometheus-operator-84d69f794d-shhvl\" (UID: \"2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-shhvl"
Jul 27 18:30:12.192931 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:12.191871    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-shhvl\" (UID: \"2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-shhvl"
Jul 27 18:30:12.278254 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:12.277502    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85d76996c4-2r2cw"]
Jul 27 18:30:12.286470 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:30:12.286428    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff718661_da68_46b2_afa6_3ef3927c722c.slice/crio-14474f9d2391d23db52e3410ebd5eba6d95943d49312d44a6d7e3b906fa907b4 WatchSource:0}: Error finding container 14474f9d2391d23db52e3410ebd5eba6d95943d49312d44a6d7e3b906fa907b4: Status 404 returned error can't find the container with id 14474f9d2391d23db52e3410ebd5eba6d95943d49312d44a6d7e3b906fa907b4
Jul 27 18:30:12.292406 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:12.292378    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-shhvl\" (UID: \"2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-shhvl"
Jul 27 18:30:12.292542 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:12.292520    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-shhvl\" (UID: \"2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-shhvl"
Jul 27 18:30:12.292639 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:12.292564    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-shhvl\" (UID: \"2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-shhvl"
Jul 27 18:30:12.292639 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:30:12.292601    2566 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 27 18:30:12.292639 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:12.292632    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn5cx\" (UniqueName: \"kubernetes.io/projected/2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50-kube-api-access-fn5cx\") pod \"prometheus-operator-84d69f794d-shhvl\" (UID: \"2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-shhvl"
Jul 27 18:30:12.292790 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:30:12.292665    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50-prometheus-operator-tls podName:2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50 nodeName:}" failed. No retries permitted until 2026-07-27 18:30:12.792640999 +0000 UTC m=+73.226898600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-shhvl" (UID: "2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50") : secret "prometheus-operator-tls" not found
Jul 27 18:30:12.293755 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:12.293709    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-shhvl\" (UID: \"2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-shhvl"
Jul 27 18:30:12.304475 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:12.304397    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-shhvl\" (UID: \"2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-shhvl"
Jul 27 18:30:12.316382 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:12.316300    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn5cx\" (UniqueName: \"kubernetes.io/projected/2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50-kube-api-access-fn5cx\") pod \"prometheus-operator-84d69f794d-shhvl\" (UID: \"2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-shhvl"
Jul 27 18:30:12.431765 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:12.430489    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qc4qw" event={"ID":"6751c8be-8944-44d2-8e25-f3fd6926225e","Type":"ContainerStarted","Data":"bdecb37397ce643f17834b8956954c4eb019b4255da8c6acbc02fdcc7b71a328"}
Jul 27 18:30:12.431765 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:12.430591    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qc4qw" event={"ID":"6751c8be-8944-44d2-8e25-f3fd6926225e","Type":"ContainerStarted","Data":"43c446cb6d7bffc3c3a0ae59f831d994a866e5145c50a53c49fe653e7f57effe"}
Jul 27 18:30:12.435405 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:12.434648    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85d76996c4-2r2cw" event={"ID":"ff718661-da68-46b2-afa6-3ef3927c722c","Type":"ContainerStarted","Data":"14474f9d2391d23db52e3410ebd5eba6d95943d49312d44a6d7e3b906fa907b4"}
Jul 27 18:30:12.798436 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:12.797556    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-shhvl\" (UID: \"2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-shhvl"
Jul 27 18:30:12.804950 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:12.804836    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-shhvl\" (UID: \"2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-shhvl"
Jul 27 18:30:13.050398 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.050313    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-shhvl"
Jul 27 18:30:13.244558 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.244468    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-shhvl"]
Jul 27 18:30:13.250940 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:30:13.250903    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7aefd4_d2b5_4a69_beb7_f2d0dbe7da50.slice/crio-1645525ffd16cb7f2727b70fe343c243983c7ebf040cb58cf99e7320a3f23860 WatchSource:0}: Error finding container 1645525ffd16cb7f2727b70fe343c243983c7ebf040cb58cf99e7320a3f23860: Status 404 returned error can't find the container with id 1645525ffd16cb7f2727b70fe343c243983c7ebf040cb58cf99e7320a3f23860
Jul 27 18:30:13.445670 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.444669    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-shhvl" event={"ID":"2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50","Type":"ContainerStarted","Data":"1645525ffd16cb7f2727b70fe343c243983c7ebf040cb58cf99e7320a3f23860"}
Jul 27 18:30:13.464504 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.463641    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qc4qw" podStartSLOduration=72.113850944 podStartE2EDuration="1m13.463618985s" totalImagesPullingTime="1.349768041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:29:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:30:13.462531958 +0000 UTC m=+73.896789571" watchObservedRunningTime="2026-07-27 18:30:13.463618985 +0000 UTC m=+73.897876597"
Jul 27 18:30:13.491190 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.490075    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76dddfb459-7kqzc"]
Jul 27 18:30:13.494556 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.494019    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:30:13.506288 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.506214    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 27 18:30:13.507470 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.507446    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76dddfb459-7kqzc"]
Jul 27 18:30:13.605155 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.605057    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a88acf50-7a59-415d-9865-d2c494231fb0-oauth-serving-cert\") pod \"console-76dddfb459-7kqzc\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") " pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:30:13.605155 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.605104    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a88acf50-7a59-415d-9865-d2c494231fb0-console-config\") pod \"console-76dddfb459-7kqzc\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") " pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:30:13.605155 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.605154    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgt6m\" (UniqueName: \"kubernetes.io/projected/a88acf50-7a59-415d-9865-d2c494231fb0-kube-api-access-wgt6m\") pod \"console-76dddfb459-7kqzc\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") " pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:30:13.605383 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.605197    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a88acf50-7a59-415d-9865-d2c494231fb0-console-serving-cert\") pod \"console-76dddfb459-7kqzc\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") " pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:30:13.605383 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.605225    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a88acf50-7a59-415d-9865-d2c494231fb0-service-ca\") pod \"console-76dddfb459-7kqzc\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") " pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:30:13.605383 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.605254    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a88acf50-7a59-415d-9865-d2c494231fb0-console-oauth-config\") pod \"console-76dddfb459-7kqzc\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") " pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:30:13.605383 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.605289    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a88acf50-7a59-415d-9865-d2c494231fb0-trusted-ca-bundle\") pod \"console-76dddfb459-7kqzc\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") " pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:30:13.706558 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.706476    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a88acf50-7a59-415d-9865-d2c494231fb0-trusted-ca-bundle\") pod \"console-76dddfb459-7kqzc\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") " pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:30:13.706558 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.706525    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a88acf50-7a59-415d-9865-d2c494231fb0-oauth-serving-cert\") pod \"console-76dddfb459-7kqzc\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") " pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:30:13.706558 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.706555    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a88acf50-7a59-415d-9865-d2c494231fb0-console-config\") pod \"console-76dddfb459-7kqzc\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") " pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:30:13.706837 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.706580    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgt6m\" (UniqueName: \"kubernetes.io/projected/a88acf50-7a59-415d-9865-d2c494231fb0-kube-api-access-wgt6m\") pod \"console-76dddfb459-7kqzc\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") " pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:30:13.706837 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.706635    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a88acf50-7a59-415d-9865-d2c494231fb0-console-serving-cert\") pod \"console-76dddfb459-7kqzc\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") " pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:30:13.706837 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.706666    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a88acf50-7a59-415d-9865-d2c494231fb0-service-ca\") pod \"console-76dddfb459-7kqzc\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") " pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:30:13.706837 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.706702    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a88acf50-7a59-415d-9865-d2c494231fb0-console-oauth-config\") pod \"console-76dddfb459-7kqzc\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") " pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:30:13.707744 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.707720    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a88acf50-7a59-415d-9865-d2c494231fb0-console-config\") pod \"console-76dddfb459-7kqzc\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") " pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:30:13.708585 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.708559    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a88acf50-7a59-415d-9865-d2c494231fb0-trusted-ca-bundle\") pod \"console-76dddfb459-7kqzc\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") " pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:30:13.708931 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.708909    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a88acf50-7a59-415d-9865-d2c494231fb0-oauth-serving-cert\") pod \"console-76dddfb459-7kqzc\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") " pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:30:13.709114 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.709096    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a88acf50-7a59-415d-9865-d2c494231fb0-service-ca\") pod \"console-76dddfb459-7kqzc\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") " pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:30:13.709880 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.709847    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a88acf50-7a59-415d-9865-d2c494231fb0-console-oauth-config\") pod \"console-76dddfb459-7kqzc\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") " pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:30:13.714637 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.714610    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a88acf50-7a59-415d-9865-d2c494231fb0-console-serving-cert\") pod \"console-76dddfb459-7kqzc\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") " pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:30:13.723673 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.723618    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgt6m\" (UniqueName: \"kubernetes.io/projected/a88acf50-7a59-415d-9865-d2c494231fb0-kube-api-access-wgt6m\") pod \"console-76dddfb459-7kqzc\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") " pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:30:13.819460 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:13.819108    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:30:22.447177 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:22.447036    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-r7j9k"
Jul 27 18:30:27.650422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:27.649937    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76dddfb459-7kqzc"]
Jul 27 18:30:27.654837 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:30:27.654807    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88acf50_7a59_415d_9865_d2c494231fb0.slice/crio-b867fc6896ef1e28529d702096c014a6d44b5ef43cc046fb54a653b720910807 WatchSource:0}: Error finding container b867fc6896ef1e28529d702096c014a6d44b5ef43cc046fb54a653b720910807: Status 404 returned error can't find the container with id b867fc6896ef1e28529d702096c014a6d44b5ef43cc046fb54a653b720910807
Jul 27 18:30:28.505283 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:28.505243    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ft289" event={"ID":"d94bb424-b152-4636-9e19-ba2a0a55ceb4","Type":"ContainerStarted","Data":"4ea0ba0758158af8ade61802ac604c7755521657e4ac881f6d63fe1bb66ead54"}
Jul 27 18:30:28.507195 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:28.507151    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85d76996c4-2r2cw" event={"ID":"ff718661-da68-46b2-afa6-3ef3927c722c","Type":"ContainerStarted","Data":"6afa3be37300b80df86b2f1f2122494bdfe3b9f67d45f6975a2540fb6a1c87ca"}
Jul 27 18:30:28.509437 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:28.509410    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-shhvl" event={"ID":"2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50","Type":"ContainerStarted","Data":"e4d7331b23571c03640cc5b4c284d829433b7700a64645141ee9d906b90e1814"}
Jul 27 18:30:28.509622 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:28.509595    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-shhvl" event={"ID":"2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50","Type":"ContainerStarted","Data":"ecdc2760a342c5187f139f7b582d5e08587e07c446aa6192b51e9ffd5cec69bd"}
Jul 27 18:30:28.512021 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:28.511996    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-b2b27" event={"ID":"0cc5553b-6803-4a64-a3b4-a4ef998a14dc","Type":"ContainerStarted","Data":"ac290c7e41f74429f683efc46bf50e16da51b8cf05453c4a9e59a0804b16f3c3"}
Jul 27 18:30:28.514737 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:28.514709    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-b2b27"
Jul 27 18:30:28.515589 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:28.515568    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qrfwk" event={"ID":"662aa4f5-23c0-42d2-8b21-270c4b33e9a5","Type":"ContainerStarted","Data":"e19f58736cb8c304684b85a3b6d0850601180739b3c24714a025d4a55b1114d3"}
Jul 27 18:30:28.517205 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:28.517186    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jpq2j" event={"ID":"53874b39-22ec-4554-ab95-643a49ee8699","Type":"ContainerStarted","Data":"30da8ca40a8f03d8052a33b8f421d6612c64f464cd05a92179a50dd0cd172b8c"}
Jul 27 18:30:28.517300 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:28.517228    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:30:28.518677 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:28.518656    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76dddfb459-7kqzc" event={"ID":"a88acf50-7a59-415d-9865-d2c494231fb0","Type":"ContainerStarted","Data":"0b10e83a007d16d563edfa5695716785c8d5879e8a15af503dbf6e3b4c080a62"}
Jul 27 18:30:28.518769 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:28.518695    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76dddfb459-7kqzc" event={"ID":"a88acf50-7a59-415d-9865-d2c494231fb0","Type":"ContainerStarted","Data":"b867fc6896ef1e28529d702096c014a6d44b5ef43cc046fb54a653b720910807"}
Jul 27 18:30:28.530713 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:28.530669    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85d76996c4-2r2cw" podStartSLOduration=2.337639679 podStartE2EDuration="17.530653526s" totalImagesPullingTime="15.193013847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:30:28.530402064 +0000 UTC m=+88.964659678" watchObservedRunningTime="2026-07-27 18:30:28.530653526 +0000 UTC m=+88.964911139"
Jul 27 18:30:28.533129 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:28.533097    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-b2b27"
Jul 27 18:30:28.550667 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:28.550596    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jpq2j" podStartSLOduration=72.020316976 podStartE2EDuration="1m28.550576346s" totalImagesPullingTime="16.53025937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:29:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:30:28.549901706 +0000 UTC m=+88.984159318" watchObservedRunningTime="2026-07-27 18:30:28.550576346 +0000 UTC m=+88.984833958"
Jul 27 18:30:28.570460 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:28.570407    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-b2b27" podStartSLOduration=1.9648793150000001 podStartE2EDuration="26.570389352s" totalImagesPullingTime="24.605510037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:30:28.569393106 +0000 UTC m=+89.003650718" watchObservedRunningTime="2026-07-27 18:30:28.570389352 +0000 UTC m=+89.004646964"
Jul 27 18:30:28.592645 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:28.592575    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-shhvl" podStartSLOduration=2.329229314 podStartE2EDuration="16.592551461s" totalImagesPullingTime="14.263322147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:30:28.590269852 +0000 UTC m=+89.024527496" watchObservedRunningTime="2026-07-27 18:30:28.592551461 +0000 UTC m=+89.026809073"
Jul 27 18:30:28.632339 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:28.632279    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76dddfb459-7kqzc" podStartSLOduration=15.632258019 podStartE2EDuration="15.632258019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:30:28.611148167 +0000 UTC m=+89.045405777" watchObservedRunningTime="2026-07-27 18:30:28.632258019 +0000 UTC m=+89.066515631"
Jul 27 18:30:28.633155 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:28.632802    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qrfwk" podStartSLOduration=71.476936208 podStartE2EDuration="1m28.632788353s" totalImagesPullingTime="17.155852145s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:29:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:30:28.631975957 +0000 UTC m=+89.066233567" watchObservedRunningTime="2026-07-27 18:30:28.632788353 +0000 UTC m=+89.067045967"
Jul 27 18:30:30.452771 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.452735    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-2cgng"]
Jul 27 18:30:30.485170 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.477556    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jb69h"]
Jul 27 18:30:30.485170 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.477884    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2cgng"
Jul 27 18:30:30.485170 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.482597    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 27 18:30:30.485170 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.482846    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 27 18:30:30.485170 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.483285    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-429nx\""
Jul 27 18:30:30.495107 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.495079    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-2cgng"]
Jul 27 18:30:30.495275 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.495151    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd"]
Jul 27 18:30:30.495275 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.495222    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.498827 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.498344    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 18:30:30.498827 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.498559    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 18:30:30.498827 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.498574    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6p8s5\""
Jul 27 18:30:30.498827 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.498801    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 18:30:30.521535 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.521504    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd"]
Jul 27 18:30:30.521666 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.521642    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd"
Jul 27 18:30:30.524660 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.524633    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-lftd9\""
Jul 27 18:30:30.526383 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.524876    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 27 18:30:30.526383 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.525155    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 27 18:30:30.526669 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.526644    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 27 18:30:30.538098 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.538074    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ffe2667-dc91-42e8-b89a-b8a06e28ecc0-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-2cgng\" (UID: \"6ffe2667-dc91-42e8-b89a-b8a06e28ecc0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2cgng"
Jul 27 18:30:30.538258 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.538141    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ffe2667-dc91-42e8-b89a-b8a06e28ecc0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-2cgng\" (UID: \"6ffe2667-dc91-42e8-b89a-b8a06e28ecc0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2cgng"
Jul 27 18:30:30.538258 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.538190    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ffe2667-dc91-42e8-b89a-b8a06e28ecc0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-2cgng\" (UID: \"6ffe2667-dc91-42e8-b89a-b8a06e28ecc0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2cgng"
Jul 27 18:30:30.538258 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.538222    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdmln\" (UniqueName: \"kubernetes.io/projected/6ffe2667-dc91-42e8-b89a-b8a06e28ecc0-kube-api-access-wdmln\") pod \"openshift-state-metrics-b8f49f5c6-2cgng\" (UID: \"6ffe2667-dc91-42e8-b89a-b8a06e28ecc0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2cgng"
Jul 27 18:30:30.639272 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.639234    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df58afda-7805-4559-a9ad-e5526362597b-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-jpfpd\" (UID: \"df58afda-7805-4559-a9ad-e5526362597b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd"
Jul 27 18:30:30.639455 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.639281    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl8kl\" (UniqueName: \"kubernetes.io/projected/df58afda-7805-4559-a9ad-e5526362597b-kube-api-access-wl8kl\") pod \"kube-state-metrics-56d9f944b4-jpfpd\" (UID: \"df58afda-7805-4559-a9ad-e5526362597b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd"
Jul 27 18:30:30.639455 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.639359    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/df58afda-7805-4559-a9ad-e5526362597b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-jpfpd\" (UID: \"df58afda-7805-4559-a9ad-e5526362597b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd"
Jul 27 18:30:30.639455 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.639425    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ffe2667-dc91-42e8-b89a-b8a06e28ecc0-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-2cgng\" (UID: \"6ffe2667-dc91-42e8-b89a-b8a06e28ecc0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2cgng"
Jul 27 18:30:30.639624 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.639456    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5e71958-e291-4c2d-ba62-31080dbe1a2d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jb69h\" (UID: \"b5e71958-e291-4c2d-ba62-31080dbe1a2d\") " pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.639624 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.639489    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ffe2667-dc91-42e8-b89a-b8a06e28ecc0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-2cgng\" (UID: \"6ffe2667-dc91-42e8-b89a-b8a06e28ecc0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2cgng"
Jul 27 18:30:30.639624 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.639514    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5e71958-e291-4c2d-ba62-31080dbe1a2d-sys\") pod \"node-exporter-jb69h\" (UID: \"b5e71958-e291-4c2d-ba62-31080dbe1a2d\") " pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.639624 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.639574    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5e71958-e291-4c2d-ba62-31080dbe1a2d-node-exporter-textfile\") pod \"node-exporter-jb69h\" (UID: \"b5e71958-e291-4c2d-ba62-31080dbe1a2d\") " pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.639810 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.639771    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ffe2667-dc91-42e8-b89a-b8a06e28ecc0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-2cgng\" (UID: \"6ffe2667-dc91-42e8-b89a-b8a06e28ecc0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2cgng"
Jul 27 18:30:30.639869 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.639839    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdmln\" (UniqueName: \"kubernetes.io/projected/6ffe2667-dc91-42e8-b89a-b8a06e28ecc0-kube-api-access-wdmln\") pod \"openshift-state-metrics-b8f49f5c6-2cgng\" (UID: \"6ffe2667-dc91-42e8-b89a-b8a06e28ecc0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2cgng"
Jul 27 18:30:30.639930 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.639875    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5e71958-e291-4c2d-ba62-31080dbe1a2d-node-exporter-wtmp\") pod \"node-exporter-jb69h\" (UID: \"b5e71958-e291-4c2d-ba62-31080dbe1a2d\") " pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.639983 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.639935    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5e71958-e291-4c2d-ba62-31080dbe1a2d-node-exporter-accelerators-collector-config\") pod \"node-exporter-jb69h\" (UID: \"b5e71958-e291-4c2d-ba62-31080dbe1a2d\") " pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.640036 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.639989    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdsjs\" (UniqueName: \"kubernetes.io/projected/b5e71958-e291-4c2d-ba62-31080dbe1a2d-kube-api-access-pdsjs\") pod \"node-exporter-jb69h\" (UID: \"b5e71958-e291-4c2d-ba62-31080dbe1a2d\") " pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.640087 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.640038    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5e71958-e291-4c2d-ba62-31080dbe1a2d-root\") pod \"node-exporter-jb69h\" (UID: \"b5e71958-e291-4c2d-ba62-31080dbe1a2d\") " pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.640087 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.640063    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/df58afda-7805-4559-a9ad-e5526362597b-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-jpfpd\" (UID: \"df58afda-7805-4559-a9ad-e5526362597b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd"
Jul 27 18:30:30.640222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.640100    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/df58afda-7805-4559-a9ad-e5526362597b-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-jpfpd\" (UID: \"df58afda-7805-4559-a9ad-e5526362597b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd"
Jul 27 18:30:30.640222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.640181    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5e71958-e291-4c2d-ba62-31080dbe1a2d-node-exporter-tls\") pod \"node-exporter-jb69h\" (UID: \"b5e71958-e291-4c2d-ba62-31080dbe1a2d\") " pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.640222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.640204    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5e71958-e291-4c2d-ba62-31080dbe1a2d-metrics-client-ca\") pod \"node-exporter-jb69h\" (UID: \"b5e71958-e291-4c2d-ba62-31080dbe1a2d\") " pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.640355 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.640232    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df58afda-7805-4559-a9ad-e5526362597b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-jpfpd\" (UID: \"df58afda-7805-4559-a9ad-e5526362597b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd"
Jul 27 18:30:30.640355 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.640250    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ffe2667-dc91-42e8-b89a-b8a06e28ecc0-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-2cgng\" (UID: \"6ffe2667-dc91-42e8-b89a-b8a06e28ecc0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2cgng"
Jul 27 18:30:30.643105 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.643082    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ffe2667-dc91-42e8-b89a-b8a06e28ecc0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-2cgng\" (UID: \"6ffe2667-dc91-42e8-b89a-b8a06e28ecc0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2cgng"
Jul 27 18:30:30.643600 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.643576    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ffe2667-dc91-42e8-b89a-b8a06e28ecc0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-2cgng\" (UID: \"6ffe2667-dc91-42e8-b89a-b8a06e28ecc0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2cgng"
Jul 27 18:30:30.652579 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.652523    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdmln\" (UniqueName: \"kubernetes.io/projected/6ffe2667-dc91-42e8-b89a-b8a06e28ecc0-kube-api-access-wdmln\") pod \"openshift-state-metrics-b8f49f5c6-2cgng\" (UID: \"6ffe2667-dc91-42e8-b89a-b8a06e28ecc0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2cgng"
Jul 27 18:30:30.741070 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.740899    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5e71958-e291-4c2d-ba62-31080dbe1a2d-node-exporter-textfile\") pod \"node-exporter-jb69h\" (UID: \"b5e71958-e291-4c2d-ba62-31080dbe1a2d\") " pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.741070 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.740970    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5e71958-e291-4c2d-ba62-31080dbe1a2d-node-exporter-wtmp\") pod \"node-exporter-jb69h\" (UID: \"b5e71958-e291-4c2d-ba62-31080dbe1a2d\") " pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.741070 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.740998    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5e71958-e291-4c2d-ba62-31080dbe1a2d-node-exporter-accelerators-collector-config\") pod \"node-exporter-jb69h\" (UID: \"b5e71958-e291-4c2d-ba62-31080dbe1a2d\") " pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.741070 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.741027    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdsjs\" (UniqueName: \"kubernetes.io/projected/b5e71958-e291-4c2d-ba62-31080dbe1a2d-kube-api-access-pdsjs\") pod \"node-exporter-jb69h\" (UID: \"b5e71958-e291-4c2d-ba62-31080dbe1a2d\") " pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.741070 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.741062    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5e71958-e291-4c2d-ba62-31080dbe1a2d-root\") pod \"node-exporter-jb69h\" (UID: \"b5e71958-e291-4c2d-ba62-31080dbe1a2d\") " pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.741493 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.741091    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/df58afda-7805-4559-a9ad-e5526362597b-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-jpfpd\" (UID: \"df58afda-7805-4559-a9ad-e5526362597b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd"
Jul 27 18:30:30.741493 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.741138    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/df58afda-7805-4559-a9ad-e5526362597b-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-jpfpd\" (UID: \"df58afda-7805-4559-a9ad-e5526362597b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd"
Jul 27 18:30:30.741493 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.741174    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5e71958-e291-4c2d-ba62-31080dbe1a2d-node-exporter-tls\") pod \"node-exporter-jb69h\" (UID: \"b5e71958-e291-4c2d-ba62-31080dbe1a2d\") " pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.741493 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.741198    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5e71958-e291-4c2d-ba62-31080dbe1a2d-metrics-client-ca\") pod \"node-exporter-jb69h\" (UID: \"b5e71958-e291-4c2d-ba62-31080dbe1a2d\") " pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.741493 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.741226    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df58afda-7805-4559-a9ad-e5526362597b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-jpfpd\" (UID: \"df58afda-7805-4559-a9ad-e5526362597b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd"
Jul 27 18:30:30.741493 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.741254    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df58afda-7805-4559-a9ad-e5526362597b-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-jpfpd\" (UID: \"df58afda-7805-4559-a9ad-e5526362597b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd"
Jul 27 18:30:30.741493 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.741278    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl8kl\" (UniqueName: \"kubernetes.io/projected/df58afda-7805-4559-a9ad-e5526362597b-kube-api-access-wl8kl\") pod \"kube-state-metrics-56d9f944b4-jpfpd\" (UID: \"df58afda-7805-4559-a9ad-e5526362597b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd"
Jul 27 18:30:30.741493 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.741311    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/df58afda-7805-4559-a9ad-e5526362597b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-jpfpd\" (UID: \"df58afda-7805-4559-a9ad-e5526362597b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd"
Jul 27 18:30:30.741493 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.741354    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5e71958-e291-4c2d-ba62-31080dbe1a2d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jb69h\" (UID: \"b5e71958-e291-4c2d-ba62-31080dbe1a2d\") " pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.741493 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.741383    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5e71958-e291-4c2d-ba62-31080dbe1a2d-sys\") pod \"node-exporter-jb69h\" (UID: \"b5e71958-e291-4c2d-ba62-31080dbe1a2d\") " pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.741493 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.741483    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5e71958-e291-4c2d-ba62-31080dbe1a2d-sys\") pod \"node-exporter-jb69h\" (UID: \"b5e71958-e291-4c2d-ba62-31080dbe1a2d\") " pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.743513 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.742110    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df58afda-7805-4559-a9ad-e5526362597b-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-jpfpd\" (UID: \"df58afda-7805-4559-a9ad-e5526362597b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd"
Jul 27 18:30:30.743513 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.742488    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/df58afda-7805-4559-a9ad-e5526362597b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-jpfpd\" (UID: \"df58afda-7805-4559-a9ad-e5526362597b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd"
Jul 27 18:30:30.743513 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.742628    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5e71958-e291-4c2d-ba62-31080dbe1a2d-node-exporter-wtmp\") pod \"node-exporter-jb69h\" (UID: \"b5e71958-e291-4c2d-ba62-31080dbe1a2d\") " pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.743513 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.742680    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5e71958-e291-4c2d-ba62-31080dbe1a2d-root\") pod \"node-exporter-jb69h\" (UID: \"b5e71958-e291-4c2d-ba62-31080dbe1a2d\") " pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.743513 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.742995    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5e71958-e291-4c2d-ba62-31080dbe1a2d-metrics-client-ca\") pod \"node-exporter-jb69h\" (UID: \"b5e71958-e291-4c2d-ba62-31080dbe1a2d\") " pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.743513 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.743063    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/df58afda-7805-4559-a9ad-e5526362597b-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-jpfpd\" (UID: \"df58afda-7805-4559-a9ad-e5526362597b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd"
Jul 27 18:30:30.743513 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:30:30.743177    2566 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 27 18:30:30.743513 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:30:30.743249    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df58afda-7805-4559-a9ad-e5526362597b-kube-state-metrics-tls podName:df58afda-7805-4559-a9ad-e5526362597b nodeName:}" failed. No retries permitted until 2026-07-27 18:30:31.243230042 +0000 UTC m=+91.677487650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/df58afda-7805-4559-a9ad-e5526362597b-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-jpfpd" (UID: "df58afda-7805-4559-a9ad-e5526362597b") : secret "kube-state-metrics-tls" not found
Jul 27 18:30:30.743513 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.743476    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5e71958-e291-4c2d-ba62-31080dbe1a2d-node-exporter-textfile\") pod \"node-exporter-jb69h\" (UID: \"b5e71958-e291-4c2d-ba62-31080dbe1a2d\") " pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.743513 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.743496    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5e71958-e291-4c2d-ba62-31080dbe1a2d-node-exporter-accelerators-collector-config\") pod \"node-exporter-jb69h\" (UID: \"b5e71958-e291-4c2d-ba62-31080dbe1a2d\") " pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.746172 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.746095    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5e71958-e291-4c2d-ba62-31080dbe1a2d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jb69h\" (UID: \"b5e71958-e291-4c2d-ba62-31080dbe1a2d\") " pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.746670 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.746624    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5e71958-e291-4c2d-ba62-31080dbe1a2d-node-exporter-tls\") pod \"node-exporter-jb69h\" (UID: \"b5e71958-e291-4c2d-ba62-31080dbe1a2d\") " pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.748641 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.748588    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df58afda-7805-4559-a9ad-e5526362597b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-jpfpd\" (UID: \"df58afda-7805-4559-a9ad-e5526362597b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd"
Jul 27 18:30:30.751356 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.751315    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl8kl\" (UniqueName: \"kubernetes.io/projected/df58afda-7805-4559-a9ad-e5526362597b-kube-api-access-wl8kl\") pod \"kube-state-metrics-56d9f944b4-jpfpd\" (UID: \"df58afda-7805-4559-a9ad-e5526362597b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd"
Jul 27 18:30:30.754995 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.754941    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdsjs\" (UniqueName: \"kubernetes.io/projected/b5e71958-e291-4c2d-ba62-31080dbe1a2d-kube-api-access-pdsjs\") pod \"node-exporter-jb69h\" (UID: \"b5e71958-e291-4c2d-ba62-31080dbe1a2d\") " pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:30.792812 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.792750    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2cgng"
Jul 27 18:30:30.808152 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:30.808093    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jb69h"
Jul 27 18:30:31.246767 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.246722    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/df58afda-7805-4559-a9ad-e5526362597b-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-jpfpd\" (UID: \"df58afda-7805-4559-a9ad-e5526362597b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd"
Jul 27 18:30:31.250142 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.250074    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/df58afda-7805-4559-a9ad-e5526362597b-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-jpfpd\" (UID: \"df58afda-7805-4559-a9ad-e5526362597b\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd"
Jul 27 18:30:31.435543 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.435509    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd"
Jul 27 18:30:31.567191 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.567158    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 18:30:31.575927 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.575808    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.579190 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.579164    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 27 18:30:31.579936 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.579396    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 27 18:30:31.579936 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.579590    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-56vtv\""
Jul 27 18:30:31.590238 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.590166    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 27 18:30:31.595615 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.590886    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 27 18:30:31.595615 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.591690    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 27 18:30:31.595615 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.592462    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 27 18:30:31.599074 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.598553    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 18:30:31.601471 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.599837    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 27 18:30:31.601471 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.600656    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 27 18:30:31.609054 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.609020    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 27 18:30:31.612371 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.611572    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85d76996c4-2r2cw"
Jul 27 18:30:31.612371 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.611636    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85d76996c4-2r2cw"
Jul 27 18:30:31.620361 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.620137    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85d76996c4-2r2cw"
Jul 27 18:30:31.650459 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.650419    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksqp5\" (UniqueName: \"kubernetes.io/projected/99005149-56be-449a-b2fe-78470e8b3e94-kube-api-access-ksqp5\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.650655 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.650470    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.650655 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.650571    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-config-volume\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.650655 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.650608    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-web-config\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.650655 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.650646    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/99005149-56be-449a-b2fe-78470e8b3e94-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.650851 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.650704    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.650851 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.650756    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.650851 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.650788    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.650851 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.650835    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/99005149-56be-449a-b2fe-78470e8b3e94-config-out\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.651050 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.650868    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99005149-56be-449a-b2fe-78470e8b3e94-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.651050 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.650893    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99005149-56be-449a-b2fe-78470e8b3e94-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.651050 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.650929    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/99005149-56be-449a-b2fe-78470e8b3e94-tls-assets\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.651050 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.650952    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.751762 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.751720    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.752020 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.751795    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.752020 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.751831    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.752020 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.751876    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/99005149-56be-449a-b2fe-78470e8b3e94-config-out\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.752020 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.751905    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99005149-56be-449a-b2fe-78470e8b3e94-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.752020 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.751926    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99005149-56be-449a-b2fe-78470e8b3e94-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.752020 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.751985    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/99005149-56be-449a-b2fe-78470e8b3e94-tls-assets\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.752020 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.752005    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.752413 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.752058    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksqp5\" (UniqueName: \"kubernetes.io/projected/99005149-56be-449a-b2fe-78470e8b3e94-kube-api-access-ksqp5\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.752413 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.752095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.752413 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.752176    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-config-volume\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.752413 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.752199    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-web-config\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.752413 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.752227    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/99005149-56be-449a-b2fe-78470e8b3e94-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.754324 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.752801    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/99005149-56be-449a-b2fe-78470e8b3e94-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.754324 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.754065    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99005149-56be-449a-b2fe-78470e8b3e94-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.754324 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.754221    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99005149-56be-449a-b2fe-78470e8b3e94-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.761319 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.761267    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.763378 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.763353    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksqp5\" (UniqueName: \"kubernetes.io/projected/99005149-56be-449a-b2fe-78470e8b3e94-kube-api-access-ksqp5\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.764102 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.763906    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.764655 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.764611    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.765098 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.765070    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/99005149-56be-449a-b2fe-78470e8b3e94-tls-assets\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.765832 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.765651    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.765832 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.765798    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/99005149-56be-449a-b2fe-78470e8b3e94-config-out\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.766221 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.766174    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-config-volume\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.766311 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.766269    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-web-config\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.766509 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.766485    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:31.902545 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:31.902458    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:30:32.430286 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:30:32.430255    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e71958_e291_4c2d_ba62_31080dbe1a2d.slice/crio-df60831e2a841fe080f7a85090b7ca74c31891dc1124eb431db5306d97052ab9 WatchSource:0}: Error finding container df60831e2a841fe080f7a85090b7ca74c31891dc1124eb431db5306d97052ab9: Status 404 returned error can't find the container with id df60831e2a841fe080f7a85090b7ca74c31891dc1124eb431db5306d97052ab9
Jul 27 18:30:32.547473 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:32.547308    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jb69h" event={"ID":"b5e71958-e291-4c2d-ba62-31080dbe1a2d","Type":"ContainerStarted","Data":"df60831e2a841fe080f7a85090b7ca74c31891dc1124eb431db5306d97052ab9"}
Jul 27 18:30:32.558764 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:32.558536    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85d76996c4-2r2cw"
Jul 27 18:30:32.643990 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:32.643679    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd"]
Jul 27 18:30:32.665910 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:32.665882    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-2cgng"]
Jul 27 18:30:32.694993 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:32.694916    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 18:30:32.839759 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:30:32.839713    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf58afda_7805_4559_a9ad_e5526362597b.slice/crio-042b6c67fb3ae6c92ad60d8e4de4b815535daa926d7acc6b51bd6815a880f403 WatchSource:0}: Error finding container 042b6c67fb3ae6c92ad60d8e4de4b815535daa926d7acc6b51bd6815a880f403: Status 404 returned error can't find the container with id 042b6c67fb3ae6c92ad60d8e4de4b815535daa926d7acc6b51bd6815a880f403
Jul 27 18:30:33.466875 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.466820    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-698c5745b9-vmg67"]
Jul 27 18:30:33.487158 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.487054    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-698c5745b9-vmg67"]
Jul 27 18:30:33.487325 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.487203    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:33.491620 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.490772    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 27 18:30:33.491620 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.491017    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 27 18:30:33.491620 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.491244    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-d59rr0ui4ene0\""
Jul 27 18:30:33.491620 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.491437    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 27 18:30:33.492000 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.491980    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 27 18:30:33.492094 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.491985    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-r69x7\""
Jul 27 18:30:33.492094 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.492026    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 27 18:30:33.557937 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.557851    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2cgng" event={"ID":"6ffe2667-dc91-42e8-b89a-b8a06e28ecc0","Type":"ContainerStarted","Data":"7f8c090f3ad7c6de4688557cd2e14a87e7f0e5582bd1ef677be9057568ba7e21"}
Jul 27 18:30:33.558155 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.557962    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2cgng" event={"ID":"6ffe2667-dc91-42e8-b89a-b8a06e28ecc0","Type":"ContainerStarted","Data":"7128e9654d0321606acd21839e4487d70aac06cb5abf6e837f3cc671e898c04f"}
Jul 27 18:30:33.558155 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.558001    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2cgng" event={"ID":"6ffe2667-dc91-42e8-b89a-b8a06e28ecc0","Type":"ContainerStarted","Data":"414eb7549f3172f9eaa381f8f05fdf2be8e23b8b4e9e404c5dc11f15f88d883d"}
Jul 27 18:30:33.559860 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.559815    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99005149-56be-449a-b2fe-78470e8b3e94","Type":"ContainerStarted","Data":"12b4bc46089a5454be3e91ae3ddbc762ddc6e21a559738d7514c35cd45b9e878"}
Jul 27 18:30:33.561644 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.561559    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd" event={"ID":"df58afda-7805-4559-a9ad-e5526362597b","Type":"ContainerStarted","Data":"042b6c67fb3ae6c92ad60d8e4de4b815535daa926d7acc6b51bd6815a880f403"}
Jul 27 18:30:33.566100 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.566076    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ft289" event={"ID":"d94bb424-b152-4636-9e19-ba2a0a55ceb4","Type":"ContainerStarted","Data":"6fefc2efbbf0212d1274dec9152bee6c316e1d89c74a416f10d8077950a9d855"}
Jul 27 18:30:33.573056 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.572806    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67d33c62-ae92-4941-b009-b77e3e93a2c2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-698c5745b9-vmg67\" (UID: \"67d33c62-ae92-4941-b009-b77e3e93a2c2\") " pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:33.573056 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.572848    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/67d33c62-ae92-4941-b009-b77e3e93a2c2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-698c5745b9-vmg67\" (UID: \"67d33c62-ae92-4941-b009-b77e3e93a2c2\") " pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:33.573056 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.572899    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/67d33c62-ae92-4941-b009-b77e3e93a2c2-secret-thanos-querier-tls\") pod \"thanos-querier-698c5745b9-vmg67\" (UID: \"67d33c62-ae92-4941-b009-b77e3e93a2c2\") " pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:33.573056 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.572926    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/67d33c62-ae92-4941-b009-b77e3e93a2c2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-698c5745b9-vmg67\" (UID: \"67d33c62-ae92-4941-b009-b77e3e93a2c2\") " pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:33.573056 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.572994    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9xgn\" (UniqueName: \"kubernetes.io/projected/67d33c62-ae92-4941-b009-b77e3e93a2c2-kube-api-access-s9xgn\") pod \"thanos-querier-698c5745b9-vmg67\" (UID: \"67d33c62-ae92-4941-b009-b77e3e93a2c2\") " pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:33.573056 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.573024    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67d33c62-ae92-4941-b009-b77e3e93a2c2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-698c5745b9-vmg67\" (UID: \"67d33c62-ae92-4941-b009-b77e3e93a2c2\") " pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:33.573056 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.573053    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/67d33c62-ae92-4941-b009-b77e3e93a2c2-secret-grpc-tls\") pod \"thanos-querier-698c5745b9-vmg67\" (UID: \"67d33c62-ae92-4941-b009-b77e3e93a2c2\") " pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:33.573490 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.573096    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67d33c62-ae92-4941-b009-b77e3e93a2c2-metrics-client-ca\") pod \"thanos-querier-698c5745b9-vmg67\" (UID: \"67d33c62-ae92-4941-b009-b77e3e93a2c2\") " pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:33.588693 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.588627    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ft289" podStartSLOduration=2.423254496 podStartE2EDuration="32.588606781s" totalImagesPullingTime="30.165352285s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:30:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:30:33.586762133 +0000 UTC m=+94.021019745" watchObservedRunningTime="2026-07-27 18:30:33.588606781 +0000 UTC m=+94.022864392"
Jul 27 18:30:33.673753 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.673714    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9xgn\" (UniqueName: \"kubernetes.io/projected/67d33c62-ae92-4941-b009-b77e3e93a2c2-kube-api-access-s9xgn\") pod \"thanos-querier-698c5745b9-vmg67\" (UID: \"67d33c62-ae92-4941-b009-b77e3e93a2c2\") " pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:33.674258 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.673774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67d33c62-ae92-4941-b009-b77e3e93a2c2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-698c5745b9-vmg67\" (UID: \"67d33c62-ae92-4941-b009-b77e3e93a2c2\") " pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:33.674258 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.673808    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/67d33c62-ae92-4941-b009-b77e3e93a2c2-secret-grpc-tls\") pod \"thanos-querier-698c5745b9-vmg67\" (UID: \"67d33c62-ae92-4941-b009-b77e3e93a2c2\") " pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:33.674258 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.673875    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67d33c62-ae92-4941-b009-b77e3e93a2c2-metrics-client-ca\") pod \"thanos-querier-698c5745b9-vmg67\" (UID: \"67d33c62-ae92-4941-b009-b77e3e93a2c2\") " pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:33.674258 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.673936    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67d33c62-ae92-4941-b009-b77e3e93a2c2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-698c5745b9-vmg67\" (UID: \"67d33c62-ae92-4941-b009-b77e3e93a2c2\") " pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:33.674258 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.673963    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/67d33c62-ae92-4941-b009-b77e3e93a2c2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-698c5745b9-vmg67\" (UID: \"67d33c62-ae92-4941-b009-b77e3e93a2c2\") " pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:33.674258 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.674049    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/67d33c62-ae92-4941-b009-b77e3e93a2c2-secret-thanos-querier-tls\") pod \"thanos-querier-698c5745b9-vmg67\" (UID: \"67d33c62-ae92-4941-b009-b77e3e93a2c2\") " pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:33.674258 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.674074    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/67d33c62-ae92-4941-b009-b77e3e93a2c2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-698c5745b9-vmg67\" (UID: \"67d33c62-ae92-4941-b009-b77e3e93a2c2\") " pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:33.677889 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.677854    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67d33c62-ae92-4941-b009-b77e3e93a2c2-metrics-client-ca\") pod \"thanos-querier-698c5745b9-vmg67\" (UID: \"67d33c62-ae92-4941-b009-b77e3e93a2c2\") " pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:33.679287 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.679239    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67d33c62-ae92-4941-b009-b77e3e93a2c2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-698c5745b9-vmg67\" (UID: \"67d33c62-ae92-4941-b009-b77e3e93a2c2\") " pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:33.688264 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.688104    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/67d33c62-ae92-4941-b009-b77e3e93a2c2-secret-grpc-tls\") pod \"thanos-querier-698c5745b9-vmg67\" (UID: \"67d33c62-ae92-4941-b009-b77e3e93a2c2\") " pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:33.688826 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.688782    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/67d33c62-ae92-4941-b009-b77e3e93a2c2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-698c5745b9-vmg67\" (UID: \"67d33c62-ae92-4941-b009-b77e3e93a2c2\") " pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:33.689373 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.689333    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/67d33c62-ae92-4941-b009-b77e3e93a2c2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-698c5745b9-vmg67\" (UID: \"67d33c62-ae92-4941-b009-b77e3e93a2c2\") " pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:33.690005 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.689957    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67d33c62-ae92-4941-b009-b77e3e93a2c2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-698c5745b9-vmg67\" (UID: \"67d33c62-ae92-4941-b009-b77e3e93a2c2\") " pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:33.691792 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.691751    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9xgn\" (UniqueName: \"kubernetes.io/projected/67d33c62-ae92-4941-b009-b77e3e93a2c2-kube-api-access-s9xgn\") pod \"thanos-querier-698c5745b9-vmg67\" (UID: \"67d33c62-ae92-4941-b009-b77e3e93a2c2\") " pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:33.693203 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.693051    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/67d33c62-ae92-4941-b009-b77e3e93a2c2-secret-thanos-querier-tls\") pod \"thanos-querier-698c5745b9-vmg67\" (UID: \"67d33c62-ae92-4941-b009-b77e3e93a2c2\") " pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:33.809376 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.809107    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:33.820794 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.820754    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:30:33.820955 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.820820    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:30:33.830405 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:33.829055    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:30:34.047243 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:34.047134    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-698c5745b9-vmg67"]
Jul 27 18:30:34.575038 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:34.574817    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:30:34.636096 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:34.636006    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85d76996c4-2r2cw"]
Jul 27 18:30:35.575289 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:35.575254    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67" event={"ID":"67d33c62-ae92-4941-b009-b77e3e93a2c2","Type":"ContainerStarted","Data":"2638d132083445f9408acccc91e69d887b02490276de3a5c5febec26211a101c"}
Jul 27 18:30:36.695414 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:36.695303    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67668d4b56-7gt4q"]
Jul 27 18:30:36.724070 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:36.724040    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67668d4b56-7gt4q"]
Jul 27 18:30:36.724202 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:36.724190    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:30:36.817141 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:36.817090    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f80c1d4-c0f2-4362-890e-36d864925a76-service-ca\") pod \"console-67668d4b56-7gt4q\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") " pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:30:36.817257 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:36.817200    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f80c1d4-c0f2-4362-890e-36d864925a76-trusted-ca-bundle\") pod \"console-67668d4b56-7gt4q\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") " pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:30:36.817257 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:36.817248    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f80c1d4-c0f2-4362-890e-36d864925a76-console-serving-cert\") pod \"console-67668d4b56-7gt4q\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") " pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:30:36.817382 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:36.817278    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f80c1d4-c0f2-4362-890e-36d864925a76-console-config\") pod \"console-67668d4b56-7gt4q\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") " pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:30:36.817382 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:36.817311    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f80c1d4-c0f2-4362-890e-36d864925a76-console-oauth-config\") pod \"console-67668d4b56-7gt4q\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") " pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:30:36.817382 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:36.817348    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8mzz\" (UniqueName: \"kubernetes.io/projected/6f80c1d4-c0f2-4362-890e-36d864925a76-kube-api-access-m8mzz\") pod \"console-67668d4b56-7gt4q\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") " pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:30:36.817540 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:36.817385    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f80c1d4-c0f2-4362-890e-36d864925a76-oauth-serving-cert\") pod \"console-67668d4b56-7gt4q\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") " pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:30:36.918762 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:36.918725    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f80c1d4-c0f2-4362-890e-36d864925a76-service-ca\") pod \"console-67668d4b56-7gt4q\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") " pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:30:36.918897 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:36.918780    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f80c1d4-c0f2-4362-890e-36d864925a76-trusted-ca-bundle\") pod \"console-67668d4b56-7gt4q\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") " pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:30:36.918897 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:36.918825    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f80c1d4-c0f2-4362-890e-36d864925a76-console-serving-cert\") pod \"console-67668d4b56-7gt4q\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") " pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:30:36.918897 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:36.918857    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f80c1d4-c0f2-4362-890e-36d864925a76-console-config\") pod \"console-67668d4b56-7gt4q\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") " pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:30:36.918897 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:36.918894    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f80c1d4-c0f2-4362-890e-36d864925a76-console-oauth-config\") pod \"console-67668d4b56-7gt4q\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") " pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:30:36.919148 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:36.918932    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8mzz\" (UniqueName: \"kubernetes.io/projected/6f80c1d4-c0f2-4362-890e-36d864925a76-kube-api-access-m8mzz\") pod \"console-67668d4b56-7gt4q\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") " pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:30:36.919148 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:36.918972    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f80c1d4-c0f2-4362-890e-36d864925a76-oauth-serving-cert\") pod \"console-67668d4b56-7gt4q\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") " pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:30:36.921769 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:36.919743    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f80c1d4-c0f2-4362-890e-36d864925a76-oauth-serving-cert\") pod \"console-67668d4b56-7gt4q\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") " pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:30:36.921769 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:36.920320    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f80c1d4-c0f2-4362-890e-36d864925a76-service-ca\") pod \"console-67668d4b56-7gt4q\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") " pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:30:36.921769 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:36.921041    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f80c1d4-c0f2-4362-890e-36d864925a76-trusted-ca-bundle\") pod \"console-67668d4b56-7gt4q\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") " pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:30:36.922003 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:36.921952    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f80c1d4-c0f2-4362-890e-36d864925a76-console-config\") pod \"console-67668d4b56-7gt4q\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") " pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:30:36.926374 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:36.926334    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f80c1d4-c0f2-4362-890e-36d864925a76-console-serving-cert\") pod \"console-67668d4b56-7gt4q\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") " pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:30:36.926667 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:36.926524    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f80c1d4-c0f2-4362-890e-36d864925a76-console-oauth-config\") pod \"console-67668d4b56-7gt4q\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") " pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:30:36.957056 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:36.956990    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8mzz\" (UniqueName: \"kubernetes.io/projected/6f80c1d4-c0f2-4362-890e-36d864925a76-kube-api-access-m8mzz\") pod \"console-67668d4b56-7gt4q\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") " pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:30:37.063763 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:37.063725    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:30:37.266954 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:37.266665    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67668d4b56-7gt4q"]
Jul 27 18:30:37.269534 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:30:37.269496    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f80c1d4_c0f2_4362_890e_36d864925a76.slice/crio-e9b4a920243b05fab512c33a0b05dff436f03b14d97a3f9b56ade8e38233b706 WatchSource:0}: Error finding container e9b4a920243b05fab512c33a0b05dff436f03b14d97a3f9b56ade8e38233b706: Status 404 returned error can't find the container with id e9b4a920243b05fab512c33a0b05dff436f03b14d97a3f9b56ade8e38233b706
Jul 27 18:30:37.594094 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:37.594062    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2cgng" event={"ID":"6ffe2667-dc91-42e8-b89a-b8a06e28ecc0","Type":"ContainerStarted","Data":"23a3ad05b9398b6c4666be87af7999030cb221fcfeedeac00c695c26fb008f61"}
Jul 27 18:30:37.596353 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:37.596327    2566 generic.go:356] "Generic (PLEG): container finished" podID="99005149-56be-449a-b2fe-78470e8b3e94" containerID="ab25df29cca025021788a28213b91390316db1e9a13470adbe70043a45090035" exitCode=0
Jul 27 18:30:37.596475 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:37.596407    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99005149-56be-449a-b2fe-78470e8b3e94","Type":"ContainerDied","Data":"ab25df29cca025021788a28213b91390316db1e9a13470adbe70043a45090035"}
Jul 27 18:30:37.598808 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:37.598785    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd" event={"ID":"df58afda-7805-4559-a9ad-e5526362597b","Type":"ContainerStarted","Data":"c89a76f94ae8e98348f0300a7ba9c9fa4bf42c2e4510177bd3182bf2725c9584"}
Jul 27 18:30:37.598908 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:37.598841    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd" event={"ID":"df58afda-7805-4559-a9ad-e5526362597b","Type":"ContainerStarted","Data":"55f636e80f82e58a26f600a1b9ca94c2129d347f55a89dac0861952886547b36"}
Jul 27 18:30:37.598908 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:37.598871    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd" event={"ID":"df58afda-7805-4559-a9ad-e5526362597b","Type":"ContainerStarted","Data":"44301c9ddfe3a3d2fae9af53bf93f17367d324e990d3a16d26ce3b234166db64"}
Jul 27 18:30:37.600395 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:37.600375    2566 generic.go:356] "Generic (PLEG): container finished" podID="b5e71958-e291-4c2d-ba62-31080dbe1a2d" containerID="d9333559796de8025eac2fd913d3bd186f6b2a389ef6432c4524a9e1c6f99a3f" exitCode=0
Jul 27 18:30:37.600471 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:37.600456    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jb69h" event={"ID":"b5e71958-e291-4c2d-ba62-31080dbe1a2d","Type":"ContainerDied","Data":"d9333559796de8025eac2fd913d3bd186f6b2a389ef6432c4524a9e1c6f99a3f"}
Jul 27 18:30:37.602155 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:37.602132    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67668d4b56-7gt4q" event={"ID":"6f80c1d4-c0f2-4362-890e-36d864925a76","Type":"ContainerStarted","Data":"1938ef3ce64b1f615d9bbf095388d297104cf4ac0fa31999857e33837a528abb"}
Jul 27 18:30:37.602229 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:37.602170    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67668d4b56-7gt4q" event={"ID":"6f80c1d4-c0f2-4362-890e-36d864925a76","Type":"ContainerStarted","Data":"e9b4a920243b05fab512c33a0b05dff436f03b14d97a3f9b56ade8e38233b706"}
Jul 27 18:30:37.614187 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:37.614138    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2cgng" podStartSLOduration=4.16328196 podStartE2EDuration="7.614106882s" totalImagesPullingTime="3.450824922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:30:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:30:37.613577212 +0000 UTC m=+98.047834824" watchObservedRunningTime="2026-07-27 18:30:37.614106882 +0000 UTC m=+98.048364493"
Jul 27 18:30:37.663673 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:37.663625    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67668d4b56-7gt4q" podStartSLOduration=1.6636086190000001 podStartE2EDuration="1.663608619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:30:37.662401101 +0000 UTC m=+98.096658711" watchObservedRunningTime="2026-07-27 18:30:37.663608619 +0000 UTC m=+98.097866208"
Jul 27 18:30:37.707869 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:37.707652    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jpfpd" podStartSLOduration=3.999561707 podStartE2EDuration="7.707630392s" totalImagesPullingTime="3.708068685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:30:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:30:37.707078361 +0000 UTC m=+98.141335973" watchObservedRunningTime="2026-07-27 18:30:37.707630392 +0000 UTC m=+98.141888006"
Jul 27 18:30:38.611719 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:38.611672    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jb69h" event={"ID":"b5e71958-e291-4c2d-ba62-31080dbe1a2d","Type":"ContainerStarted","Data":"31be384f30f67c8b3c63f80e1b95c8669581213a4c4c7ec71853df82748cdedf"}
Jul 27 18:30:38.611966 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:38.611741    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jb69h" event={"ID":"b5e71958-e291-4c2d-ba62-31080dbe1a2d","Type":"ContainerStarted","Data":"ffef5d90686034ed32024252591b3e6a6debd4adea2e5c563029562edacba8fc"}
Jul 27 18:30:38.643194 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:38.643136    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jb69h" podStartSLOduration=4.380429767 podStartE2EDuration="8.643091844s" totalImagesPullingTime="4.140936616s" totalInitContainerRuntime="121.725461ms" isStatefulPod=true podCreationTimestamp="2026-07-27 18:30:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:30:38.640213464 +0000 UTC m=+99.074471077" watchObservedRunningTime="2026-07-27 18:30:38.643091844 +0000 UTC m=+99.077349456"
Jul 27 18:30:39.621949 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:39.621910    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67" event={"ID":"67d33c62-ae92-4941-b009-b77e3e93a2c2","Type":"ContainerStarted","Data":"726a3983cf4783565b91bbe5553a180f0e48d38fec3071870b00b35ad851d30c"}
Jul 27 18:30:39.622532 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:39.622003    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67" event={"ID":"67d33c62-ae92-4941-b009-b77e3e93a2c2","Type":"ContainerStarted","Data":"96e92fbbb6a64db2a0d4d509d0ba3611b73a274996b7ea199fb0afff45b27fd2"}
Jul 27 18:30:39.622532 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:39.622031    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67" event={"ID":"67d33c62-ae92-4941-b009-b77e3e93a2c2","Type":"ContainerStarted","Data":"28ac9424047d375e083113079122602ba71df9e62b9498f044dc2dca895a792c"}
Jul 27 18:30:40.628376 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:40.628342    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99005149-56be-449a-b2fe-78470e8b3e94","Type":"ContainerStarted","Data":"3c45420d3b8c0ccae1ec28efa5be655b9db1be02cbc006559517a58f3824e8bd"}
Jul 27 18:30:40.628823 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:40.628415    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99005149-56be-449a-b2fe-78470e8b3e94","Type":"ContainerStarted","Data":"abc3cece2957cf83867e8caacd7a09c711341d3df76a6298729fa8e7cbef82f0"}
Jul 27 18:30:40.628823 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:40.628442    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99005149-56be-449a-b2fe-78470e8b3e94","Type":"ContainerStarted","Data":"d38261ac7012567a81d7922e1ef936a13497414d0c0193d3a3eeb08d9e6b5749"}
Jul 27 18:30:40.628823 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:40.628466    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99005149-56be-449a-b2fe-78470e8b3e94","Type":"ContainerStarted","Data":"cfd94a139e3795ead6f9c3b54e165b43e5c881d98b14b02e4f50508b3449c5d8"}
Jul 27 18:30:40.628823 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:40.628489    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99005149-56be-449a-b2fe-78470e8b3e94","Type":"ContainerStarted","Data":"18019c08f6dd1c4785ce30ee257d498c3a987952923a54005a4fe9f094b9acd6"}
Jul 27 18:30:42.640512 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:42.640460    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99005149-56be-449a-b2fe-78470e8b3e94","Type":"ContainerStarted","Data":"b1f632393ae55b945299ec3211aaed26d6776d9e1c699e0585ce19c145b91292"}
Jul 27 18:30:42.643393 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:42.643369    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67" event={"ID":"67d33c62-ae92-4941-b009-b77e3e93a2c2","Type":"ContainerStarted","Data":"baf05e2105b2b3aa59d11bb79b229a43be623b3ece6b938d4f8fb326eff4274b"}
Jul 27 18:30:42.643523 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:42.643421    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67" event={"ID":"67d33c62-ae92-4941-b009-b77e3e93a2c2","Type":"ContainerStarted","Data":"806a9f931c506a67783a280712599825b314d5cfab01d857c08b9999b59ce357"}
Jul 27 18:30:42.643523 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:42.643447    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67" event={"ID":"67d33c62-ae92-4941-b009-b77e3e93a2c2","Type":"ContainerStarted","Data":"133c9a37f81ca4803aa05973f85ac02d7d76183679d507436cffd10ca13b99f1"}
Jul 27 18:30:42.643630 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:42.643570    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:42.701094 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:42.701044    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.232352293 podStartE2EDuration="11.701021887s" totalImagesPullingTime="7.437639416s" totalInitContainerRuntime="31.030178ms" isStatefulPod=false podCreationTimestamp="2026-07-27 18:30:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:30:42.676083499 +0000 UTC m=+103.110341132" watchObservedRunningTime="2026-07-27 18:30:42.701021887 +0000 UTC m=+103.135279500"
Jul 27 18:30:42.701296 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:42.701227    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67" podStartSLOduration=3.377697727 podStartE2EDuration="9.701217641s" totalImagesPullingTime="6.323519914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:30:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:30:42.698455755 +0000 UTC m=+103.132713367" watchObservedRunningTime="2026-07-27 18:30:42.701217641 +0000 UTC m=+103.135475254"
Jul 27 18:30:43.652647 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:43.652623    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-698c5745b9-vmg67"
Jul 27 18:30:47.064482 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:47.064441    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:30:47.064909 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:47.064503    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:30:47.069133 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:47.069093    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:30:47.668364 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:47.668330    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:30:47.722220 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:47.722187    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76dddfb459-7kqzc"]
Jul 27 18:30:59.525876 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:59.525838    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jpq2j"
Jul 27 18:30:59.665134 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:59.665076    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-85d76996c4-2r2cw" podUID="ff718661-da68-46b2-afa6-3ef3927c722c" containerName="console" containerID="cri-o://6afa3be37300b80df86b2f1f2122494bdfe3b9f67d45f6975a2540fb6a1c87ca" gracePeriod=15
Jul 27 18:30:59.939410 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:30:59.939378    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85d76996c4-2r2cw"
Jul 27 18:31:00.012672 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:00.012632    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff718661-da68-46b2-afa6-3ef3927c722c-console-config\") pod \"ff718661-da68-46b2-afa6-3ef3927c722c\" (UID: \"ff718661-da68-46b2-afa6-3ef3927c722c\") "
Jul 27 18:31:00.012860 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:00.012681    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff718661-da68-46b2-afa6-3ef3927c722c-oauth-serving-cert\") pod \"ff718661-da68-46b2-afa6-3ef3927c722c\" (UID: \"ff718661-da68-46b2-afa6-3ef3927c722c\") "
Jul 27 18:31:00.012860 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:00.012718    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff718661-da68-46b2-afa6-3ef3927c722c-console-oauth-config\") pod \"ff718661-da68-46b2-afa6-3ef3927c722c\" (UID: \"ff718661-da68-46b2-afa6-3ef3927c722c\") "
Jul 27 18:31:00.012860 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:00.012741    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pp8sm\" (UniqueName: \"kubernetes.io/projected/ff718661-da68-46b2-afa6-3ef3927c722c-kube-api-access-pp8sm\") pod \"ff718661-da68-46b2-afa6-3ef3927c722c\" (UID: \"ff718661-da68-46b2-afa6-3ef3927c722c\") "
Jul 27 18:31:00.012860 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:00.012759    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff718661-da68-46b2-afa6-3ef3927c722c-service-ca\") pod \"ff718661-da68-46b2-afa6-3ef3927c722c\" (UID: \"ff718661-da68-46b2-afa6-3ef3927c722c\") "
Jul 27 18:31:00.012860 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:00.012791    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff718661-da68-46b2-afa6-3ef3927c722c-console-serving-cert\") pod \"ff718661-da68-46b2-afa6-3ef3927c722c\" (UID: \"ff718661-da68-46b2-afa6-3ef3927c722c\") "
Jul 27 18:31:00.013195 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:00.013171    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff718661-da68-46b2-afa6-3ef3927c722c-console-config" (OuterVolumeSpecName: "console-config") pod "ff718661-da68-46b2-afa6-3ef3927c722c" (UID: "ff718661-da68-46b2-afa6-3ef3927c722c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:31:00.013253 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:00.013218    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff718661-da68-46b2-afa6-3ef3927c722c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ff718661-da68-46b2-afa6-3ef3927c722c" (UID: "ff718661-da68-46b2-afa6-3ef3927c722c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:31:00.013289 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:00.013235    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff718661-da68-46b2-afa6-3ef3927c722c-service-ca" (OuterVolumeSpecName: "service-ca") pod "ff718661-da68-46b2-afa6-3ef3927c722c" (UID: "ff718661-da68-46b2-afa6-3ef3927c722c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:31:00.015351 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:00.015319    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff718661-da68-46b2-afa6-3ef3927c722c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ff718661-da68-46b2-afa6-3ef3927c722c" (UID: "ff718661-da68-46b2-afa6-3ef3927c722c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:31:00.015446 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:00.015404    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff718661-da68-46b2-afa6-3ef3927c722c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ff718661-da68-46b2-afa6-3ef3927c722c" (UID: "ff718661-da68-46b2-afa6-3ef3927c722c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:31:00.015446 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:00.015409    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff718661-da68-46b2-afa6-3ef3927c722c-kube-api-access-pp8sm" (OuterVolumeSpecName: "kube-api-access-pp8sm") pod "ff718661-da68-46b2-afa6-3ef3927c722c" (UID: "ff718661-da68-46b2-afa6-3ef3927c722c"). InnerVolumeSpecName "kube-api-access-pp8sm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:31:00.113800 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:00.113713    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff718661-da68-46b2-afa6-3ef3927c722c-oauth-serving-cert\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:31:00.113800 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:00.113744    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff718661-da68-46b2-afa6-3ef3927c722c-console-oauth-config\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:31:00.113800 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:00.113753    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp8sm\" (UniqueName: \"kubernetes.io/projected/ff718661-da68-46b2-afa6-3ef3927c722c-kube-api-access-pp8sm\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:31:00.113800 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:00.113762    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff718661-da68-46b2-afa6-3ef3927c722c-service-ca\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:31:00.113800 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:00.113769    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff718661-da68-46b2-afa6-3ef3927c722c-console-serving-cert\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:31:00.113800 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:00.113778    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff718661-da68-46b2-afa6-3ef3927c722c-console-config\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:31:00.720100 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:00.720066    2566 generic.go:356] "Generic (PLEG): container finished" podID="ff718661-da68-46b2-afa6-3ef3927c722c" containerID="6afa3be37300b80df86b2f1f2122494bdfe3b9f67d45f6975a2540fb6a1c87ca" exitCode=0
Jul 27 18:31:00.720564 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:00.720138    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85d76996c4-2r2cw"
Jul 27 18:31:00.720564 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:00.720164    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85d76996c4-2r2cw" event={"ID":"ff718661-da68-46b2-afa6-3ef3927c722c","Type":"ContainerDied","Data":"6afa3be37300b80df86b2f1f2122494bdfe3b9f67d45f6975a2540fb6a1c87ca"}
Jul 27 18:31:00.720564 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:00.720228    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85d76996c4-2r2cw" event={"ID":"ff718661-da68-46b2-afa6-3ef3927c722c","Type":"ContainerDied","Data":"14474f9d2391d23db52e3410ebd5eba6d95943d49312d44a6d7e3b906fa907b4"}
Jul 27 18:31:00.720564 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:00.720260    2566 scope.go:117] "RemoveContainer" containerID="6afa3be37300b80df86b2f1f2122494bdfe3b9f67d45f6975a2540fb6a1c87ca"
Jul 27 18:31:00.728567 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:00.728551    2566 scope.go:117] "RemoveContainer" containerID="6afa3be37300b80df86b2f1f2122494bdfe3b9f67d45f6975a2540fb6a1c87ca"
Jul 27 18:31:00.728808 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:31:00.728786    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6afa3be37300b80df86b2f1f2122494bdfe3b9f67d45f6975a2540fb6a1c87ca\": container with ID starting with 6afa3be37300b80df86b2f1f2122494bdfe3b9f67d45f6975a2540fb6a1c87ca not found: ID does not exist" containerID="6afa3be37300b80df86b2f1f2122494bdfe3b9f67d45f6975a2540fb6a1c87ca"
Jul 27 18:31:00.728876 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:00.728821    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6afa3be37300b80df86b2f1f2122494bdfe3b9f67d45f6975a2540fb6a1c87ca"} err="failed to get container status \"6afa3be37300b80df86b2f1f2122494bdfe3b9f67d45f6975a2540fb6a1c87ca\": rpc error: code = NotFound desc = could not find container \"6afa3be37300b80df86b2f1f2122494bdfe3b9f67d45f6975a2540fb6a1c87ca\": container with ID starting with 6afa3be37300b80df86b2f1f2122494bdfe3b9f67d45f6975a2540fb6a1c87ca not found: ID does not exist"
Jul 27 18:31:00.742787 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:00.742761    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85d76996c4-2r2cw"]
Jul 27 18:31:00.747906 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:00.747879    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85d76996c4-2r2cw"]
Jul 27 18:31:02.123109 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:02.123076    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff718661-da68-46b2-afa6-3ef3927c722c" path="/var/lib/kubelet/pods/ff718661-da68-46b2-afa6-3ef3927c722c/volumes"
Jul 27 18:31:12.741743 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:12.741666    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-76dddfb459-7kqzc" podUID="a88acf50-7a59-415d-9865-d2c494231fb0" containerName="console" containerID="cri-o://0b10e83a007d16d563edfa5695716785c8d5879e8a15af503dbf6e3b4c080a62" gracePeriod=15
Jul 27 18:31:13.529656 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.529627    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:31:13.631991 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.631961    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a88acf50-7a59-415d-9865-d2c494231fb0-console-config\") pod \"a88acf50-7a59-415d-9865-d2c494231fb0\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") "
Jul 27 18:31:13.632194 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.632037    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgt6m\" (UniqueName: \"kubernetes.io/projected/a88acf50-7a59-415d-9865-d2c494231fb0-kube-api-access-wgt6m\") pod \"a88acf50-7a59-415d-9865-d2c494231fb0\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") "
Jul 27 18:31:13.632194 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.632086    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a88acf50-7a59-415d-9865-d2c494231fb0-oauth-serving-cert\") pod \"a88acf50-7a59-415d-9865-d2c494231fb0\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") "
Jul 27 18:31:13.632194 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.632146    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a88acf50-7a59-415d-9865-d2c494231fb0-service-ca\") pod \"a88acf50-7a59-415d-9865-d2c494231fb0\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") "
Jul 27 18:31:13.632194 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.632184    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a88acf50-7a59-415d-9865-d2c494231fb0-console-oauth-config\") pod \"a88acf50-7a59-415d-9865-d2c494231fb0\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") "
Jul 27 18:31:13.632460 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.632240    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a88acf50-7a59-415d-9865-d2c494231fb0-trusted-ca-bundle\") pod \"a88acf50-7a59-415d-9865-d2c494231fb0\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") "
Jul 27 18:31:13.632460 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.632278    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a88acf50-7a59-415d-9865-d2c494231fb0-console-serving-cert\") pod \"a88acf50-7a59-415d-9865-d2c494231fb0\" (UID: \"a88acf50-7a59-415d-9865-d2c494231fb0\") "
Jul 27 18:31:13.632460 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.632399    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a88acf50-7a59-415d-9865-d2c494231fb0-console-config" (OuterVolumeSpecName: "console-config") pod "a88acf50-7a59-415d-9865-d2c494231fb0" (UID: "a88acf50-7a59-415d-9865-d2c494231fb0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:31:13.632614 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.632513    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a88acf50-7a59-415d-9865-d2c494231fb0-console-config\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:31:13.632614 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.632550    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a88acf50-7a59-415d-9865-d2c494231fb0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a88acf50-7a59-415d-9865-d2c494231fb0" (UID: "a88acf50-7a59-415d-9865-d2c494231fb0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:31:13.632694 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.632615    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a88acf50-7a59-415d-9865-d2c494231fb0-service-ca" (OuterVolumeSpecName: "service-ca") pod "a88acf50-7a59-415d-9865-d2c494231fb0" (UID: "a88acf50-7a59-415d-9865-d2c494231fb0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:31:13.632694 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.632654    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a88acf50-7a59-415d-9865-d2c494231fb0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a88acf50-7a59-415d-9865-d2c494231fb0" (UID: "a88acf50-7a59-415d-9865-d2c494231fb0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:31:13.634602 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.634548    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a88acf50-7a59-415d-9865-d2c494231fb0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a88acf50-7a59-415d-9865-d2c494231fb0" (UID: "a88acf50-7a59-415d-9865-d2c494231fb0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:31:13.634602 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.634555    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a88acf50-7a59-415d-9865-d2c494231fb0-kube-api-access-wgt6m" (OuterVolumeSpecName: "kube-api-access-wgt6m") pod "a88acf50-7a59-415d-9865-d2c494231fb0" (UID: "a88acf50-7a59-415d-9865-d2c494231fb0"). InnerVolumeSpecName "kube-api-access-wgt6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:31:13.634707 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.634623    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a88acf50-7a59-415d-9865-d2c494231fb0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a88acf50-7a59-415d-9865-d2c494231fb0" (UID: "a88acf50-7a59-415d-9865-d2c494231fb0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:31:13.733904 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.733873    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgt6m\" (UniqueName: \"kubernetes.io/projected/a88acf50-7a59-415d-9865-d2c494231fb0-kube-api-access-wgt6m\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:31:13.733904 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.733900    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a88acf50-7a59-415d-9865-d2c494231fb0-oauth-serving-cert\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:31:13.733904 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.733915    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a88acf50-7a59-415d-9865-d2c494231fb0-service-ca\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:31:13.734164 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.733924    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a88acf50-7a59-415d-9865-d2c494231fb0-console-oauth-config\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:31:13.734164 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.733932    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a88acf50-7a59-415d-9865-d2c494231fb0-trusted-ca-bundle\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:31:13.734164 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.733941    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a88acf50-7a59-415d-9865-d2c494231fb0-console-serving-cert\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:31:13.789867 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.789832    2566 generic.go:356] "Generic (PLEG): container finished" podID="a88acf50-7a59-415d-9865-d2c494231fb0" containerID="0b10e83a007d16d563edfa5695716785c8d5879e8a15af503dbf6e3b4c080a62" exitCode=0
Jul 27 18:31:13.790274 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.789884    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76dddfb459-7kqzc"
Jul 27 18:31:13.790274 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.789918    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76dddfb459-7kqzc" event={"ID":"a88acf50-7a59-415d-9865-d2c494231fb0","Type":"ContainerDied","Data":"0b10e83a007d16d563edfa5695716785c8d5879e8a15af503dbf6e3b4c080a62"}
Jul 27 18:31:13.790274 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.789992    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76dddfb459-7kqzc" event={"ID":"a88acf50-7a59-415d-9865-d2c494231fb0","Type":"ContainerDied","Data":"b867fc6896ef1e28529d702096c014a6d44b5ef43cc046fb54a653b720910807"}
Jul 27 18:31:13.790274 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.790012    2566 scope.go:117] "RemoveContainer" containerID="0b10e83a007d16d563edfa5695716785c8d5879e8a15af503dbf6e3b4c080a62"
Jul 27 18:31:13.799183 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.799167    2566 scope.go:117] "RemoveContainer" containerID="0b10e83a007d16d563edfa5695716785c8d5879e8a15af503dbf6e3b4c080a62"
Jul 27 18:31:13.799430 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:31:13.799412    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b10e83a007d16d563edfa5695716785c8d5879e8a15af503dbf6e3b4c080a62\": container with ID starting with 0b10e83a007d16d563edfa5695716785c8d5879e8a15af503dbf6e3b4c080a62 not found: ID does not exist" containerID="0b10e83a007d16d563edfa5695716785c8d5879e8a15af503dbf6e3b4c080a62"
Jul 27 18:31:13.799478 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.799449    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b10e83a007d16d563edfa5695716785c8d5879e8a15af503dbf6e3b4c080a62"} err="failed to get container status \"0b10e83a007d16d563edfa5695716785c8d5879e8a15af503dbf6e3b4c080a62\": rpc error: code = NotFound desc = could not find container \"0b10e83a007d16d563edfa5695716785c8d5879e8a15af503dbf6e3b4c080a62\": container with ID starting with 0b10e83a007d16d563edfa5695716785c8d5879e8a15af503dbf6e3b4c080a62 not found: ID does not exist"
Jul 27 18:31:13.810697 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.810674    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76dddfb459-7kqzc"]
Jul 27 18:31:13.814010 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:13.813988    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76dddfb459-7kqzc"]
Jul 27 18:31:14.122960 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:14.122932    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a88acf50-7a59-415d-9865-d2c494231fb0" path="/var/lib/kubelet/pods/a88acf50-7a59-415d-9865-d2c494231fb0/volumes"
Jul 27 18:31:50.773414 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:50.773387    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 18:31:50.774207 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:50.773733    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="99005149-56be-449a-b2fe-78470e8b3e94" containerName="alertmanager" containerID="cri-o://18019c08f6dd1c4785ce30ee257d498c3a987952923a54005a4fe9f094b9acd6" gracePeriod=120
Jul 27 18:31:50.774207 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:50.773803    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="99005149-56be-449a-b2fe-78470e8b3e94" containerName="kube-rbac-proxy-metric" containerID="cri-o://3c45420d3b8c0ccae1ec28efa5be655b9db1be02cbc006559517a58f3824e8bd" gracePeriod=120
Jul 27 18:31:50.774207 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:50.773827    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="99005149-56be-449a-b2fe-78470e8b3e94" containerName="kube-rbac-proxy-web" containerID="cri-o://d38261ac7012567a81d7922e1ef936a13497414d0c0193d3a3eeb08d9e6b5749" gracePeriod=120
Jul 27 18:31:50.774207 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:50.773868    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="99005149-56be-449a-b2fe-78470e8b3e94" containerName="prom-label-proxy" containerID="cri-o://b1f632393ae55b945299ec3211aaed26d6776d9e1c699e0585ce19c145b91292" gracePeriod=120
Jul 27 18:31:50.774207 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:50.773934    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="99005149-56be-449a-b2fe-78470e8b3e94" containerName="kube-rbac-proxy" containerID="cri-o://abc3cece2957cf83867e8caacd7a09c711341d3df76a6298729fa8e7cbef82f0" gracePeriod=120
Jul 27 18:31:50.774207 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:50.774044    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="99005149-56be-449a-b2fe-78470e8b3e94" containerName="config-reloader" containerID="cri-o://cfd94a139e3795ead6f9c3b54e165b43e5c881d98b14b02e4f50508b3449c5d8" gracePeriod=120
Jul 27 18:31:50.936130 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:50.936085    2566 generic.go:356] "Generic (PLEG): container finished" podID="99005149-56be-449a-b2fe-78470e8b3e94" containerID="b1f632393ae55b945299ec3211aaed26d6776d9e1c699e0585ce19c145b91292" exitCode=0
Jul 27 18:31:50.936130 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:50.936109    2566 generic.go:356] "Generic (PLEG): container finished" podID="99005149-56be-449a-b2fe-78470e8b3e94" containerID="3c45420d3b8c0ccae1ec28efa5be655b9db1be02cbc006559517a58f3824e8bd" exitCode=0
Jul 27 18:31:50.936130 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:50.936133    2566 generic.go:356] "Generic (PLEG): container finished" podID="99005149-56be-449a-b2fe-78470e8b3e94" containerID="abc3cece2957cf83867e8caacd7a09c711341d3df76a6298729fa8e7cbef82f0" exitCode=0
Jul 27 18:31:50.936357 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:50.936145    2566 generic.go:356] "Generic (PLEG): container finished" podID="99005149-56be-449a-b2fe-78470e8b3e94" containerID="cfd94a139e3795ead6f9c3b54e165b43e5c881d98b14b02e4f50508b3449c5d8" exitCode=0
Jul 27 18:31:50.936357 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:50.936146    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99005149-56be-449a-b2fe-78470e8b3e94","Type":"ContainerDied","Data":"b1f632393ae55b945299ec3211aaed26d6776d9e1c699e0585ce19c145b91292"}
Jul 27 18:31:50.936357 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:50.936215    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99005149-56be-449a-b2fe-78470e8b3e94","Type":"ContainerDied","Data":"3c45420d3b8c0ccae1ec28efa5be655b9db1be02cbc006559517a58f3824e8bd"}
Jul 27 18:31:50.936357 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:50.936155    2566 generic.go:356] "Generic (PLEG): container finished" podID="99005149-56be-449a-b2fe-78470e8b3e94" containerID="18019c08f6dd1c4785ce30ee257d498c3a987952923a54005a4fe9f094b9acd6" exitCode=0
Jul 27 18:31:50.936357 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:50.936243    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99005149-56be-449a-b2fe-78470e8b3e94","Type":"ContainerDied","Data":"abc3cece2957cf83867e8caacd7a09c711341d3df76a6298729fa8e7cbef82f0"}
Jul 27 18:31:50.936357 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:50.936266    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99005149-56be-449a-b2fe-78470e8b3e94","Type":"ContainerDied","Data":"cfd94a139e3795ead6f9c3b54e165b43e5c881d98b14b02e4f50508b3449c5d8"}
Jul 27 18:31:50.936357 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:50.936289    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99005149-56be-449a-b2fe-78470e8b3e94","Type":"ContainerDied","Data":"18019c08f6dd1c4785ce30ee257d498c3a987952923a54005a4fe9f094b9acd6"}
Jul 27 18:31:51.945881 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:51.945852    2566 generic.go:356] "Generic (PLEG): container finished" podID="99005149-56be-449a-b2fe-78470e8b3e94" containerID="d38261ac7012567a81d7922e1ef936a13497414d0c0193d3a3eeb08d9e6b5749" exitCode=0
Jul 27 18:31:51.946321 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:51.945912    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99005149-56be-449a-b2fe-78470e8b3e94","Type":"ContainerDied","Data":"d38261ac7012567a81d7922e1ef936a13497414d0c0193d3a3eeb08d9e6b5749"}
Jul 27 18:31:52.040032 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.040008    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:52.040384 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.040366    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-secret-alertmanager-main-tls\") pod \"99005149-56be-449a-b2fe-78470e8b3e94\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") "
Jul 27 18:31:52.040430 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.040394    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-web-config\") pod \"99005149-56be-449a-b2fe-78470e8b3e94\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") "
Jul 27 18:31:52.040430 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.040417    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-cluster-tls-config\") pod \"99005149-56be-449a-b2fe-78470e8b3e94\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") "
Jul 27 18:31:52.040496 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.040442    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-secret-alertmanager-kube-rbac-proxy-metric\") pod \"99005149-56be-449a-b2fe-78470e8b3e94\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") "
Jul 27 18:31:52.040496 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.040472    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-secret-alertmanager-kube-rbac-proxy\") pod \"99005149-56be-449a-b2fe-78470e8b3e94\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") "
Jul 27 18:31:52.040587 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.040496    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-secret-alertmanager-kube-rbac-proxy-web\") pod \"99005149-56be-449a-b2fe-78470e8b3e94\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") "
Jul 27 18:31:52.040587 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.040528    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99005149-56be-449a-b2fe-78470e8b3e94-metrics-client-ca\") pod \"99005149-56be-449a-b2fe-78470e8b3e94\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") "
Jul 27 18:31:52.040587 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.040552    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/99005149-56be-449a-b2fe-78470e8b3e94-tls-assets\") pod \"99005149-56be-449a-b2fe-78470e8b3e94\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") "
Jul 27 18:31:52.040587 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.040583    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-config-volume\") pod \"99005149-56be-449a-b2fe-78470e8b3e94\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") "
Jul 27 18:31:52.040788 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.040608    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99005149-56be-449a-b2fe-78470e8b3e94-alertmanager-trusted-ca-bundle\") pod \"99005149-56be-449a-b2fe-78470e8b3e94\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") "
Jul 27 18:31:52.040788 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.040649    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/99005149-56be-449a-b2fe-78470e8b3e94-alertmanager-main-db\") pod \"99005149-56be-449a-b2fe-78470e8b3e94\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") "
Jul 27 18:31:52.040788 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.040687    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksqp5\" (UniqueName: \"kubernetes.io/projected/99005149-56be-449a-b2fe-78470e8b3e94-kube-api-access-ksqp5\") pod \"99005149-56be-449a-b2fe-78470e8b3e94\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") "
Jul 27 18:31:52.040788 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.040711    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/99005149-56be-449a-b2fe-78470e8b3e94-config-out\") pod \"99005149-56be-449a-b2fe-78470e8b3e94\" (UID: \"99005149-56be-449a-b2fe-78470e8b3e94\") "
Jul 27 18:31:52.041043 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.040951    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99005149-56be-449a-b2fe-78470e8b3e94-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "99005149-56be-449a-b2fe-78470e8b3e94" (UID: "99005149-56be-449a-b2fe-78470e8b3e94"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:31:52.042190 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.042161    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99005149-56be-449a-b2fe-78470e8b3e94-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "99005149-56be-449a-b2fe-78470e8b3e94" (UID: "99005149-56be-449a-b2fe-78470e8b3e94"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:31:52.042309 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.042287    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99005149-56be-449a-b2fe-78470e8b3e94-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "99005149-56be-449a-b2fe-78470e8b3e94" (UID: "99005149-56be-449a-b2fe-78470e8b3e94"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:31:52.043453 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.043421    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "99005149-56be-449a-b2fe-78470e8b3e94" (UID: "99005149-56be-449a-b2fe-78470e8b3e94"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:31:52.044438 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.044396    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "99005149-56be-449a-b2fe-78470e8b3e94" (UID: "99005149-56be-449a-b2fe-78470e8b3e94"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:31:52.044438 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.044428    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99005149-56be-449a-b2fe-78470e8b3e94-config-out" (OuterVolumeSpecName: "config-out") pod "99005149-56be-449a-b2fe-78470e8b3e94" (UID: "99005149-56be-449a-b2fe-78470e8b3e94"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:31:52.044789 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.044759    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-config-volume" (OuterVolumeSpecName: "config-volume") pod "99005149-56be-449a-b2fe-78470e8b3e94" (UID: "99005149-56be-449a-b2fe-78470e8b3e94"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:31:52.044872 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.044847    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99005149-56be-449a-b2fe-78470e8b3e94-kube-api-access-ksqp5" (OuterVolumeSpecName: "kube-api-access-ksqp5") pod "99005149-56be-449a-b2fe-78470e8b3e94" (UID: "99005149-56be-449a-b2fe-78470e8b3e94"). InnerVolumeSpecName "kube-api-access-ksqp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:31:52.044994 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.044969    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99005149-56be-449a-b2fe-78470e8b3e94-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "99005149-56be-449a-b2fe-78470e8b3e94" (UID: "99005149-56be-449a-b2fe-78470e8b3e94"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:31:52.045277 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.045253    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "99005149-56be-449a-b2fe-78470e8b3e94" (UID: "99005149-56be-449a-b2fe-78470e8b3e94"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:31:52.045380 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.045360    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "99005149-56be-449a-b2fe-78470e8b3e94" (UID: "99005149-56be-449a-b2fe-78470e8b3e94"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:31:52.048168 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.048140    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "99005149-56be-449a-b2fe-78470e8b3e94" (UID: "99005149-56be-449a-b2fe-78470e8b3e94"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:31:52.055617 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.055591    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-web-config" (OuterVolumeSpecName: "web-config") pod "99005149-56be-449a-b2fe-78470e8b3e94" (UID: "99005149-56be-449a-b2fe-78470e8b3e94"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:31:52.141365 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.141337    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-secret-alertmanager-main-tls\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:31:52.141365 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.141363    2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-web-config\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:31:52.141576 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.141378    2566 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-cluster-tls-config\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:31:52.141576 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.141391    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:31:52.141576 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.141407    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:31:52.141576 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.141420    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:31:52.141576 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.141434    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99005149-56be-449a-b2fe-78470e8b3e94-metrics-client-ca\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:31:52.141576 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.141446    2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/99005149-56be-449a-b2fe-78470e8b3e94-tls-assets\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:31:52.141576 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.141458    2566 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/99005149-56be-449a-b2fe-78470e8b3e94-config-volume\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:31:52.141576 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.141469    2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99005149-56be-449a-b2fe-78470e8b3e94-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:31:52.141576 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.141483    2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/99005149-56be-449a-b2fe-78470e8b3e94-alertmanager-main-db\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:31:52.141576 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.141497    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksqp5\" (UniqueName: \"kubernetes.io/projected/99005149-56be-449a-b2fe-78470e8b3e94-kube-api-access-ksqp5\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:31:52.141576 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.141512    2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/99005149-56be-449a-b2fe-78470e8b3e94-config-out\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:31:52.953093 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.953057    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"99005149-56be-449a-b2fe-78470e8b3e94","Type":"ContainerDied","Data":"12b4bc46089a5454be3e91ae3ddbc762ddc6e21a559738d7514c35cd45b9e878"}
Jul 27 18:31:52.953093 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.953074    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:52.953679 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.953149    2566 scope.go:117] "RemoveContainer" containerID="b1f632393ae55b945299ec3211aaed26d6776d9e1c699e0585ce19c145b91292"
Jul 27 18:31:52.961369 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.961351    2566 scope.go:117] "RemoveContainer" containerID="3c45420d3b8c0ccae1ec28efa5be655b9db1be02cbc006559517a58f3824e8bd"
Jul 27 18:31:52.968534 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.968520    2566 scope.go:117] "RemoveContainer" containerID="abc3cece2957cf83867e8caacd7a09c711341d3df76a6298729fa8e7cbef82f0"
Jul 27 18:31:52.977286 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.977262    2566 scope.go:117] "RemoveContainer" containerID="d38261ac7012567a81d7922e1ef936a13497414d0c0193d3a3eeb08d9e6b5749"
Jul 27 18:31:52.980957 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.980935    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 18:31:52.984802 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.984783    2566 scope.go:117] "RemoveContainer" containerID="cfd94a139e3795ead6f9c3b54e165b43e5c881d98b14b02e4f50508b3449c5d8"
Jul 27 18:31:52.990732 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.990706    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 18:31:52.992404 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.992386    2566 scope.go:117] "RemoveContainer" containerID="18019c08f6dd1c4785ce30ee257d498c3a987952923a54005a4fe9f094b9acd6"
Jul 27 18:31:52.999550 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:52.999532    2566 scope.go:117] "RemoveContainer" containerID="ab25df29cca025021788a28213b91390316db1e9a13470adbe70043a45090035"
Jul 27 18:31:53.017707 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.017683    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 18:31:53.017960 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.017948    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99005149-56be-449a-b2fe-78470e8b3e94" containerName="kube-rbac-proxy-web"
Jul 27 18:31:53.018008 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.017961    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="99005149-56be-449a-b2fe-78470e8b3e94" containerName="kube-rbac-proxy-web"
Jul 27 18:31:53.018008 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.017973    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99005149-56be-449a-b2fe-78470e8b3e94" containerName="kube-rbac-proxy-metric"
Jul 27 18:31:53.018008 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.017979    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="99005149-56be-449a-b2fe-78470e8b3e94" containerName="kube-rbac-proxy-metric"
Jul 27 18:31:53.018008 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.017988    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99005149-56be-449a-b2fe-78470e8b3e94" containerName="config-reloader"
Jul 27 18:31:53.018008 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.017993    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="99005149-56be-449a-b2fe-78470e8b3e94" containerName="config-reloader"
Jul 27 18:31:53.018008 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.018004    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99005149-56be-449a-b2fe-78470e8b3e94" containerName="alertmanager"
Jul 27 18:31:53.018222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.018012    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="99005149-56be-449a-b2fe-78470e8b3e94" containerName="alertmanager"
Jul 27 18:31:53.018222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.018020    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99005149-56be-449a-b2fe-78470e8b3e94" containerName="kube-rbac-proxy"
Jul 27 18:31:53.018222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.018028    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="99005149-56be-449a-b2fe-78470e8b3e94" containerName="kube-rbac-proxy"
Jul 27 18:31:53.018222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.018035    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff718661-da68-46b2-afa6-3ef3927c722c" containerName="console"
Jul 27 18:31:53.018222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.018040    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff718661-da68-46b2-afa6-3ef3927c722c" containerName="console"
Jul 27 18:31:53.018222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.018050    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99005149-56be-449a-b2fe-78470e8b3e94" containerName="init-config-reloader"
Jul 27 18:31:53.018222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.018062    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="99005149-56be-449a-b2fe-78470e8b3e94" containerName="init-config-reloader"
Jul 27 18:31:53.018222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.018071    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a88acf50-7a59-415d-9865-d2c494231fb0" containerName="console"
Jul 27 18:31:53.018222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.018076    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a88acf50-7a59-415d-9865-d2c494231fb0" containerName="console"
Jul 27 18:31:53.018222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.018081    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99005149-56be-449a-b2fe-78470e8b3e94" containerName="prom-label-proxy"
Jul 27 18:31:53.018222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.018086    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="99005149-56be-449a-b2fe-78470e8b3e94" containerName="prom-label-proxy"
Jul 27 18:31:53.018222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.018152    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff718661-da68-46b2-afa6-3ef3927c722c" containerName="console"
Jul 27 18:31:53.018222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.018161    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="99005149-56be-449a-b2fe-78470e8b3e94" containerName="kube-rbac-proxy"
Jul 27 18:31:53.018222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.018166    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a88acf50-7a59-415d-9865-d2c494231fb0" containerName="console"
Jul 27 18:31:53.018222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.018172    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="99005149-56be-449a-b2fe-78470e8b3e94" containerName="alertmanager"
Jul 27 18:31:53.018222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.018176    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="99005149-56be-449a-b2fe-78470e8b3e94" containerName="kube-rbac-proxy-web"
Jul 27 18:31:53.018222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.018182    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="99005149-56be-449a-b2fe-78470e8b3e94" containerName="config-reloader"
Jul 27 18:31:53.018222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.018187    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="99005149-56be-449a-b2fe-78470e8b3e94" containerName="kube-rbac-proxy-metric"
Jul 27 18:31:53.018222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.018192    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="99005149-56be-449a-b2fe-78470e8b3e94" containerName="prom-label-proxy"
Jul 27 18:31:53.022160 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.022142    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.025701 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.025682    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 27 18:31:53.025810 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.025681    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 27 18:31:53.025810 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.025682    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 27 18:31:53.026077 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.026051    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 27 18:31:53.026249 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.026226    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-56vtv\""
Jul 27 18:31:53.026322 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.026253    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 27 18:31:53.026322 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.026240    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 27 18:31:53.026540 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.026526    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 27 18:31:53.026726 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.026709    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 27 18:31:53.032440 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.032419    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 27 18:31:53.034653 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.034633    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 18:31:53.047894 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.047864    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.047894 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.047893    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-config-volume\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.048085 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.047909    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.048085 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.047939    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-config-out\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.048085 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.047954    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.048085 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.048027    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.048085 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.048063    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.048291 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.048103    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-web-config\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.048291 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.048157    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk7dp\" (UniqueName: \"kubernetes.io/projected/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-kube-api-access-nk7dp\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.048291 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.048174    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.048291 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.048192    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.048291 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.048211    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.048291 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.048227    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.148893 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.148797    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk7dp\" (UniqueName: \"kubernetes.io/projected/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-kube-api-access-nk7dp\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.148893 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.148841    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.148893 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.148861    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.148893 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.148884    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.149250 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.148907    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.149250 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.148942    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.149250 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.148967    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-config-volume\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.149250 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.148991    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.149250 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.149034    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-config-out\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.149250 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.149054    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.149250 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.149095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.149250 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.149143    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.149250 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.149180    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-web-config\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.149680 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.149472    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.149739 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.149723    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.150163 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.150022    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.152248 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.152224    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-config-out\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.152545 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.152509    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-web-config\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.152635 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.152521    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-config-volume\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.152728 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.152710    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.152791 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.152762    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.153058 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.153041    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.153161 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.153139    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.153621 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.153595    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.153919 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.153901    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.159948 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.159931    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk7dp\" (UniqueName: \"kubernetes.io/projected/bd1fb1e4-c04c-45bc-91ce-4df135cb07f0-kube-api-access-nk7dp\") pod \"alertmanager-main-0\" (UID: \"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.332063 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.332002    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 18:31:53.466729 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.466705    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 18:31:53.468797 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:31:53.468772    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd1fb1e4_c04c_45bc_91ce_4df135cb07f0.slice/crio-52559b017cfae9e59dc28747217fc97afb90733bdd653c359c7968419d38c26e WatchSource:0}: Error finding container 52559b017cfae9e59dc28747217fc97afb90733bdd653c359c7968419d38c26e: Status 404 returned error can't find the container with id 52559b017cfae9e59dc28747217fc97afb90733bdd653c359c7968419d38c26e
Jul 27 18:31:53.958386 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.958355    2566 generic.go:356] "Generic (PLEG): container finished" podID="bd1fb1e4-c04c-45bc-91ce-4df135cb07f0" containerID="d3df6538608ae464f413b7b86c26ef9a266d3ba145ae437468c9cd651ad74d4f" exitCode=0
Jul 27 18:31:53.958802 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.958440    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0","Type":"ContainerDied","Data":"d3df6538608ae464f413b7b86c26ef9a266d3ba145ae437468c9cd651ad74d4f"}
Jul 27 18:31:53.958802 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:53.958508    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0","Type":"ContainerStarted","Data":"52559b017cfae9e59dc28747217fc97afb90733bdd653c359c7968419d38c26e"}
Jul 27 18:31:54.123209 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.123181    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99005149-56be-449a-b2fe-78470e8b3e94" path="/var/lib/kubelet/pods/99005149-56be-449a-b2fe-78470e8b3e94/volumes"
Jul 27 18:31:54.783186 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.783153    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"]
Jul 27 18:31:54.786319 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.786293    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"
Jul 27 18:31:54.789242 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.789219    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 27 18:31:54.789363 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.789230    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 27 18:31:54.789363 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.789219    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 27 18:31:54.789363 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.789280    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 27 18:31:54.789363 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.789316    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-mtx29\""
Jul 27 18:31:54.789588 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.789544    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 27 18:31:54.796382 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.796349    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 27 18:31:54.798326 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.798302    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"]
Jul 27 18:31:54.865187 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.865149    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ebc1e23-4641-4e18-ac64-a2b7c605492e-serving-certs-ca-bundle\") pod \"telemeter-client-5588ddd8b5-6dg78\" (UID: \"6ebc1e23-4641-4e18-ac64-a2b7c605492e\") " pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"
Jul 27 18:31:54.865348 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.865216    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6ebc1e23-4641-4e18-ac64-a2b7c605492e-secret-telemeter-client\") pod \"telemeter-client-5588ddd8b5-6dg78\" (UID: \"6ebc1e23-4641-4e18-ac64-a2b7c605492e\") " pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"
Jul 27 18:31:54.865348 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.865321    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6ebc1e23-4641-4e18-ac64-a2b7c605492e-federate-client-tls\") pod \"telemeter-client-5588ddd8b5-6dg78\" (UID: \"6ebc1e23-4641-4e18-ac64-a2b7c605492e\") " pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"
Jul 27 18:31:54.865636 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.865354    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ebc1e23-4641-4e18-ac64-a2b7c605492e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5588ddd8b5-6dg78\" (UID: \"6ebc1e23-4641-4e18-ac64-a2b7c605492e\") " pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"
Jul 27 18:31:54.865636 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.865373    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h4l4\" (UniqueName: \"kubernetes.io/projected/6ebc1e23-4641-4e18-ac64-a2b7c605492e-kube-api-access-4h4l4\") pod \"telemeter-client-5588ddd8b5-6dg78\" (UID: \"6ebc1e23-4641-4e18-ac64-a2b7c605492e\") " pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"
Jul 27 18:31:54.865636 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.865392    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ebc1e23-4641-4e18-ac64-a2b7c605492e-metrics-client-ca\") pod \"telemeter-client-5588ddd8b5-6dg78\" (UID: \"6ebc1e23-4641-4e18-ac64-a2b7c605492e\") " pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"
Jul 27 18:31:54.865636 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.865448    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ebc1e23-4641-4e18-ac64-a2b7c605492e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5588ddd8b5-6dg78\" (UID: \"6ebc1e23-4641-4e18-ac64-a2b7c605492e\") " pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"
Jul 27 18:31:54.865636 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.865480    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6ebc1e23-4641-4e18-ac64-a2b7c605492e-telemeter-client-tls\") pod \"telemeter-client-5588ddd8b5-6dg78\" (UID: \"6ebc1e23-4641-4e18-ac64-a2b7c605492e\") " pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"
Jul 27 18:31:54.967338 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.967302    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ebc1e23-4641-4e18-ac64-a2b7c605492e-serving-certs-ca-bundle\") pod \"telemeter-client-5588ddd8b5-6dg78\" (UID: \"6ebc1e23-4641-4e18-ac64-a2b7c605492e\") " pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"
Jul 27 18:31:54.967872 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.967848    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6ebc1e23-4641-4e18-ac64-a2b7c605492e-secret-telemeter-client\") pod \"telemeter-client-5588ddd8b5-6dg78\" (UID: \"6ebc1e23-4641-4e18-ac64-a2b7c605492e\") " pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"
Jul 27 18:31:54.968034 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.968019    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6ebc1e23-4641-4e18-ac64-a2b7c605492e-federate-client-tls\") pod \"telemeter-client-5588ddd8b5-6dg78\" (UID: \"6ebc1e23-4641-4e18-ac64-a2b7c605492e\") " pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"
Jul 27 18:31:54.968160 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.968142    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ebc1e23-4641-4e18-ac64-a2b7c605492e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5588ddd8b5-6dg78\" (UID: \"6ebc1e23-4641-4e18-ac64-a2b7c605492e\") " pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"
Jul 27 18:31:54.968267 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.968251    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h4l4\" (UniqueName: \"kubernetes.io/projected/6ebc1e23-4641-4e18-ac64-a2b7c605492e-kube-api-access-4h4l4\") pod \"telemeter-client-5588ddd8b5-6dg78\" (UID: \"6ebc1e23-4641-4e18-ac64-a2b7c605492e\") " pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"
Jul 27 18:31:54.968374 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.968360    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ebc1e23-4641-4e18-ac64-a2b7c605492e-metrics-client-ca\") pod \"telemeter-client-5588ddd8b5-6dg78\" (UID: \"6ebc1e23-4641-4e18-ac64-a2b7c605492e\") " pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"
Jul 27 18:31:54.968491 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.968475    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ebc1e23-4641-4e18-ac64-a2b7c605492e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5588ddd8b5-6dg78\" (UID: \"6ebc1e23-4641-4e18-ac64-a2b7c605492e\") " pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"
Jul 27 18:31:54.968613 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.968597    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6ebc1e23-4641-4e18-ac64-a2b7c605492e-telemeter-client-tls\") pod \"telemeter-client-5588ddd8b5-6dg78\" (UID: \"6ebc1e23-4641-4e18-ac64-a2b7c605492e\") " pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"
Jul 27 18:31:54.970884 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.970856    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ebc1e23-4641-4e18-ac64-a2b7c605492e-metrics-client-ca\") pod \"telemeter-client-5588ddd8b5-6dg78\" (UID: \"6ebc1e23-4641-4e18-ac64-a2b7c605492e\") " pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"
Jul 27 18:31:54.971171 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.971142    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ebc1e23-4641-4e18-ac64-a2b7c605492e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5588ddd8b5-6dg78\" (UID: \"6ebc1e23-4641-4e18-ac64-a2b7c605492e\") " pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"
Jul 27 18:31:54.971264 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.971247    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ebc1e23-4641-4e18-ac64-a2b7c605492e-serving-certs-ca-bundle\") pod \"telemeter-client-5588ddd8b5-6dg78\" (UID: \"6ebc1e23-4641-4e18-ac64-a2b7c605492e\") " pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"
Jul 27 18:31:54.974985 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.973245    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6ebc1e23-4641-4e18-ac64-a2b7c605492e-federate-client-tls\") pod \"telemeter-client-5588ddd8b5-6dg78\" (UID: \"6ebc1e23-4641-4e18-ac64-a2b7c605492e\") " pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"
Jul 27 18:31:54.974985 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.974380    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6ebc1e23-4641-4e18-ac64-a2b7c605492e-telemeter-client-tls\") pod \"telemeter-client-5588ddd8b5-6dg78\" (UID: \"6ebc1e23-4641-4e18-ac64-a2b7c605492e\") " pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"
Jul 27 18:31:54.975740 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.975714    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ebc1e23-4641-4e18-ac64-a2b7c605492e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5588ddd8b5-6dg78\" (UID: \"6ebc1e23-4641-4e18-ac64-a2b7c605492e\") " pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"
Jul 27 18:31:54.976490 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.976466    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6ebc1e23-4641-4e18-ac64-a2b7c605492e-secret-telemeter-client\") pod \"telemeter-client-5588ddd8b5-6dg78\" (UID: \"6ebc1e23-4641-4e18-ac64-a2b7c605492e\") " pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"
Jul 27 18:31:54.977890 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.977868    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0","Type":"ContainerStarted","Data":"28f8a4c1955754dd9763f7b896088b7e8a6d3d5fabb1359ede05a5caf079c88a"}
Jul 27 18:31:54.977956 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.977946    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0","Type":"ContainerStarted","Data":"84baf0c21c67868d26279bb9c54107b407245e05cd0d7c544a085f848256f102"}
Jul 27 18:31:54.978003 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.977973    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0","Type":"ContainerStarted","Data":"d74ac9161022381d457677adaefadcc2f45fcd7c2197bc7d0a47cf636dd49040"}
Jul 27 18:31:54.978003 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.977996    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0","Type":"ContainerStarted","Data":"ce731f113b686ddcf91ca7c6e439ac18b32e56d73d0dab0a06e0843cc657ff6c"}
Jul 27 18:31:54.978075 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.978030    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0","Type":"ContainerStarted","Data":"e936679d085aa04c7aa08ea645bd74dd1a452dcca29307602abf6b6a146a4d20"}
Jul 27 18:31:54.978075 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.978065    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd1fb1e4-c04c-45bc-91ce-4df135cb07f0","Type":"ContainerStarted","Data":"b86657c9cfab1bcdc7bf9ac71ed9cb80eba9cc4105d98084d0c7f3a97040d773"}
Jul 27 18:31:54.979390 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:54.979372    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h4l4\" (UniqueName: \"kubernetes.io/projected/6ebc1e23-4641-4e18-ac64-a2b7c605492e-kube-api-access-4h4l4\") pod \"telemeter-client-5588ddd8b5-6dg78\" (UID: \"6ebc1e23-4641-4e18-ac64-a2b7c605492e\") " pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"
Jul 27 18:31:55.008644 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:55.008588    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.988739558 podStartE2EDuration="2.008570683s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.831125ms" isStatefulPod=false podCreationTimestamp="2026-07-27 18:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:31:55.006942498 +0000 UTC m=+175.441200109" watchObservedRunningTime="2026-07-27 18:31:55.008570683 +0000 UTC m=+175.442828293"
Jul 27 18:31:55.098310 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:55.098193    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"
Jul 27 18:31:55.256611 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:55.256588    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5588ddd8b5-6dg78"]
Jul 27 18:31:55.258423 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:31:55.258393    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebc1e23_4641_4e18_ac64_a2b7c605492e.slice/crio-b8d4331dcdb95884cf85db31f9ecc00f792e2b4ee7c508106cb5b5b5cd2296a0 WatchSource:0}: Error finding container b8d4331dcdb95884cf85db31f9ecc00f792e2b4ee7c508106cb5b5b5cd2296a0: Status 404 returned error can't find the container with id b8d4331dcdb95884cf85db31f9ecc00f792e2b4ee7c508106cb5b5b5cd2296a0
Jul 27 18:31:55.982895 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:55.982858    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78" event={"ID":"6ebc1e23-4641-4e18-ac64-a2b7c605492e","Type":"ContainerStarted","Data":"b8d4331dcdb95884cf85db31f9ecc00f792e2b4ee7c508106cb5b5b5cd2296a0"}
Jul 27 18:31:57.992891 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:57.992788    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78" event={"ID":"6ebc1e23-4641-4e18-ac64-a2b7c605492e","Type":"ContainerStarted","Data":"42de6718f521892ee3fb6fd0704fee0a997ce982beb54543f08cd80e76bcbe32"}
Jul 27 18:31:57.992891 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:57.992843    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78" event={"ID":"6ebc1e23-4641-4e18-ac64-a2b7c605492e","Type":"ContainerStarted","Data":"dfd69f377efef80af381fd03808985eaf55a38afdcb42cfe9b3c988677346224"}
Jul 27 18:31:57.992891 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:57.992865    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78" event={"ID":"6ebc1e23-4641-4e18-ac64-a2b7c605492e","Type":"ContainerStarted","Data":"8e6c3448427681cee07cae822d52342c51be8f62ee483a8c4474e43860478ed5"}
Jul 27 18:31:58.019679 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:58.019621    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5588ddd8b5-6dg78" podStartSLOduration=1.75486389 podStartE2EDuration="4.019606219s" totalImagesPullingTime="2.264742329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:31:58.01815313 +0000 UTC m=+178.452410742" watchObservedRunningTime="2026-07-27 18:31:58.019606219 +0000 UTC m=+178.453863830"
Jul 27 18:31:58.844087 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:58.844051    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c487c5dc9-2g97l"]
Jul 27 18:31:58.846729 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:58.846705    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:31:58.857134 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:58.857082    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c487c5dc9-2g97l"]
Jul 27 18:31:58.900900 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:58.900865    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzlc4\" (UniqueName: \"kubernetes.io/projected/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-kube-api-access-lzlc4\") pod \"console-6c487c5dc9-2g97l\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") " pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:31:58.900900 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:58.900910    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-console-oauth-config\") pod \"console-6c487c5dc9-2g97l\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") " pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:31:58.901176 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:58.900958    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-console-config\") pod \"console-6c487c5dc9-2g97l\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") " pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:31:58.901176 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:58.900977    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-oauth-serving-cert\") pod \"console-6c487c5dc9-2g97l\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") " pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:31:58.901176 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:58.901010    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-service-ca\") pod \"console-6c487c5dc9-2g97l\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") " pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:31:58.901176 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:58.901029    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-trusted-ca-bundle\") pod \"console-6c487c5dc9-2g97l\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") " pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:31:58.901176 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:58.901051    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-console-serving-cert\") pod \"console-6c487c5dc9-2g97l\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") " pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:31:59.001955 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:59.001910    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzlc4\" (UniqueName: \"kubernetes.io/projected/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-kube-api-access-lzlc4\") pod \"console-6c487c5dc9-2g97l\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") " pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:31:59.002450 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:59.001976    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-console-oauth-config\") pod \"console-6c487c5dc9-2g97l\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") " pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:31:59.002450 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:59.002023    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-console-config\") pod \"console-6c487c5dc9-2g97l\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") " pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:31:59.002450 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:59.002049    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-oauth-serving-cert\") pod \"console-6c487c5dc9-2g97l\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") " pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:31:59.002450 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:59.002095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-service-ca\") pod \"console-6c487c5dc9-2g97l\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") " pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:31:59.002450 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:59.002137    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-trusted-ca-bundle\") pod \"console-6c487c5dc9-2g97l\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") " pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:31:59.002450 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:59.002179    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-console-serving-cert\") pod \"console-6c487c5dc9-2g97l\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") " pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:31:59.002908 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:59.002883    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-oauth-serving-cert\") pod \"console-6c487c5dc9-2g97l\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") " pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:31:59.003016 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:59.002919    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-service-ca\") pod \"console-6c487c5dc9-2g97l\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") " pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:31:59.003324 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:59.003301    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-trusted-ca-bundle\") pod \"console-6c487c5dc9-2g97l\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") " pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:31:59.003897 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:59.003877    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-console-config\") pod \"console-6c487c5dc9-2g97l\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") " pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:31:59.004966 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:59.004946    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-console-oauth-config\") pod \"console-6c487c5dc9-2g97l\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") " pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:31:59.005144 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:59.005110    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-console-serving-cert\") pod \"console-6c487c5dc9-2g97l\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") " pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:31:59.011180 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:59.011159    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzlc4\" (UniqueName: \"kubernetes.io/projected/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-kube-api-access-lzlc4\") pod \"console-6c487c5dc9-2g97l\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") " pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:31:59.158080 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:59.157972    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:31:59.304829 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:31:59.304773    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c487c5dc9-2g97l"]
Jul 27 18:31:59.306968 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:31:59.306931    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ba6f74_1b4d_4427_a1a6_29de11bfd2f4.slice/crio-7128d92c780c90b6b97e9005ab8466457cfc409275f8555b9b6a9006543f0744 WatchSource:0}: Error finding container 7128d92c780c90b6b97e9005ab8466457cfc409275f8555b9b6a9006543f0744: Status 404 returned error can't find the container with id 7128d92c780c90b6b97e9005ab8466457cfc409275f8555b9b6a9006543f0744
Jul 27 18:32:00.002935 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:00.002899    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c487c5dc9-2g97l" event={"ID":"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4","Type":"ContainerStarted","Data":"30181192308cc49028547297cb21bf4e348bb3463b20004ee93984c57891cb66"}
Jul 27 18:32:00.003311 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:00.002950    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c487c5dc9-2g97l" event={"ID":"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4","Type":"ContainerStarted","Data":"7128d92c780c90b6b97e9005ab8466457cfc409275f8555b9b6a9006543f0744"}
Jul 27 18:32:00.025134 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:00.025060    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c487c5dc9-2g97l" podStartSLOduration=2.025041805 podStartE2EDuration="2.025041805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:32:00.022799057 +0000 UTC m=+180.457056672" watchObservedRunningTime="2026-07-27 18:32:00.025041805 +0000 UTC m=+180.459299418"
Jul 27 18:32:09.159324 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:09.159283    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:32:09.159708 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:09.159355    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:32:09.164040 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:09.164018    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:32:10.045811 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:10.045785    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:32:10.092993 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:10.092957    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67668d4b56-7gt4q"]
Jul 27 18:32:35.113172 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:35.113095    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-67668d4b56-7gt4q" podUID="6f80c1d4-c0f2-4362-890e-36d864925a76" containerName="console" containerID="cri-o://1938ef3ce64b1f615d9bbf095388d297104cf4ac0fa31999857e33837a528abb" gracePeriod=15
Jul 27 18:32:35.349217 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:35.347917    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:32:35.418527 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:35.418435    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f80c1d4-c0f2-4362-890e-36d864925a76-trusted-ca-bundle\") pod \"6f80c1d4-c0f2-4362-890e-36d864925a76\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") "
Jul 27 18:32:35.418527 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:35.418473    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f80c1d4-c0f2-4362-890e-36d864925a76-console-oauth-config\") pod \"6f80c1d4-c0f2-4362-890e-36d864925a76\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") "
Jul 27 18:32:35.418527 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:35.418498    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8mzz\" (UniqueName: \"kubernetes.io/projected/6f80c1d4-c0f2-4362-890e-36d864925a76-kube-api-access-m8mzz\") pod \"6f80c1d4-c0f2-4362-890e-36d864925a76\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") "
Jul 27 18:32:35.418812 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:35.418570    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f80c1d4-c0f2-4362-890e-36d864925a76-console-config\") pod \"6f80c1d4-c0f2-4362-890e-36d864925a76\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") "
Jul 27 18:32:35.418812 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:35.418603    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f80c1d4-c0f2-4362-890e-36d864925a76-oauth-serving-cert\") pod \"6f80c1d4-c0f2-4362-890e-36d864925a76\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") "
Jul 27 18:32:35.418812 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:35.418701    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f80c1d4-c0f2-4362-890e-36d864925a76-service-ca\") pod \"6f80c1d4-c0f2-4362-890e-36d864925a76\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") "
Jul 27 18:32:35.418812 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:35.418775    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f80c1d4-c0f2-4362-890e-36d864925a76-console-serving-cert\") pod \"6f80c1d4-c0f2-4362-890e-36d864925a76\" (UID: \"6f80c1d4-c0f2-4362-890e-36d864925a76\") "
Jul 27 18:32:35.419007 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:35.418921    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f80c1d4-c0f2-4362-890e-36d864925a76-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6f80c1d4-c0f2-4362-890e-36d864925a76" (UID: "6f80c1d4-c0f2-4362-890e-36d864925a76"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:32:35.419007 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:35.418965    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f80c1d4-c0f2-4362-890e-36d864925a76-console-config" (OuterVolumeSpecName: "console-config") pod "6f80c1d4-c0f2-4362-890e-36d864925a76" (UID: "6f80c1d4-c0f2-4362-890e-36d864925a76"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:32:35.419007 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:35.418970    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f80c1d4-c0f2-4362-890e-36d864925a76-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6f80c1d4-c0f2-4362-890e-36d864925a76" (UID: "6f80c1d4-c0f2-4362-890e-36d864925a76"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:32:35.419247 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:35.419224    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f80c1d4-c0f2-4362-890e-36d864925a76-console-config\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:32:35.419529 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:35.419253    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f80c1d4-c0f2-4362-890e-36d864925a76-oauth-serving-cert\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:32:35.419529 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:35.419267    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f80c1d4-c0f2-4362-890e-36d864925a76-trusted-ca-bundle\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:32:35.419529 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:35.419224    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f80c1d4-c0f2-4362-890e-36d864925a76-service-ca" (OuterVolumeSpecName: "service-ca") pod "6f80c1d4-c0f2-4362-890e-36d864925a76" (UID: "6f80c1d4-c0f2-4362-890e-36d864925a76"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:32:35.420735 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:35.420713    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f80c1d4-c0f2-4362-890e-36d864925a76-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6f80c1d4-c0f2-4362-890e-36d864925a76" (UID: "6f80c1d4-c0f2-4362-890e-36d864925a76"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:32:35.420936 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:35.420912    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f80c1d4-c0f2-4362-890e-36d864925a76-kube-api-access-m8mzz" (OuterVolumeSpecName: "kube-api-access-m8mzz") pod "6f80c1d4-c0f2-4362-890e-36d864925a76" (UID: "6f80c1d4-c0f2-4362-890e-36d864925a76"). InnerVolumeSpecName "kube-api-access-m8mzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:32:35.421130 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:35.421103    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f80c1d4-c0f2-4362-890e-36d864925a76-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6f80c1d4-c0f2-4362-890e-36d864925a76" (UID: "6f80c1d4-c0f2-4362-890e-36d864925a76"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:32:35.520593 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:35.520563    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f80c1d4-c0f2-4362-890e-36d864925a76-service-ca\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:32:35.520593 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:35.520589    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f80c1d4-c0f2-4362-890e-36d864925a76-console-serving-cert\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:32:35.520593 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:35.520599    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f80c1d4-c0f2-4362-890e-36d864925a76-console-oauth-config\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:32:35.520819 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:35.520610    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8mzz\" (UniqueName: \"kubernetes.io/projected/6f80c1d4-c0f2-4362-890e-36d864925a76-kube-api-access-m8mzz\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:32:36.139233 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:36.139203    2566 generic.go:356] "Generic (PLEG): container finished" podID="6f80c1d4-c0f2-4362-890e-36d864925a76" containerID="1938ef3ce64b1f615d9bbf095388d297104cf4ac0fa31999857e33837a528abb" exitCode=0
Jul 27 18:32:36.139645 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:36.139249    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67668d4b56-7gt4q"
Jul 27 18:32:36.139645 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:36.139294    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67668d4b56-7gt4q" event={"ID":"6f80c1d4-c0f2-4362-890e-36d864925a76","Type":"ContainerDied","Data":"1938ef3ce64b1f615d9bbf095388d297104cf4ac0fa31999857e33837a528abb"}
Jul 27 18:32:36.139645 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:36.139352    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67668d4b56-7gt4q" event={"ID":"6f80c1d4-c0f2-4362-890e-36d864925a76","Type":"ContainerDied","Data":"e9b4a920243b05fab512c33a0b05dff436f03b14d97a3f9b56ade8e38233b706"}
Jul 27 18:32:36.139645 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:36.139384    2566 scope.go:117] "RemoveContainer" containerID="1938ef3ce64b1f615d9bbf095388d297104cf4ac0fa31999857e33837a528abb"
Jul 27 18:32:36.147541 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:36.147524    2566 scope.go:117] "RemoveContainer" containerID="1938ef3ce64b1f615d9bbf095388d297104cf4ac0fa31999857e33837a528abb"
Jul 27 18:32:36.147790 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:32:36.147771    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1938ef3ce64b1f615d9bbf095388d297104cf4ac0fa31999857e33837a528abb\": container with ID starting with 1938ef3ce64b1f615d9bbf095388d297104cf4ac0fa31999857e33837a528abb not found: ID does not exist" containerID="1938ef3ce64b1f615d9bbf095388d297104cf4ac0fa31999857e33837a528abb"
Jul 27 18:32:36.147841 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:36.147798    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1938ef3ce64b1f615d9bbf095388d297104cf4ac0fa31999857e33837a528abb"} err="failed to get container status \"1938ef3ce64b1f615d9bbf095388d297104cf4ac0fa31999857e33837a528abb\": rpc error: code = NotFound desc = could not find container \"1938ef3ce64b1f615d9bbf095388d297104cf4ac0fa31999857e33837a528abb\": container with ID starting with 1938ef3ce64b1f615d9bbf095388d297104cf4ac0fa31999857e33837a528abb not found: ID does not exist"
Jul 27 18:32:36.157374 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:36.157342    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67668d4b56-7gt4q"]
Jul 27 18:32:36.161037 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:36.161015    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67668d4b56-7gt4q"]
Jul 27 18:32:38.122765 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:32:38.122732    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f80c1d4-c0f2-4362-890e-36d864925a76" path="/var/lib/kubelet/pods/6f80c1d4-c0f2-4362-890e-36d864925a76/volumes"
Jul 27 18:33:09.408491 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.408454    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58cf7c967-q62dt"]
Jul 27 18:33:09.408895 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.408805    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f80c1d4-c0f2-4362-890e-36d864925a76" containerName="console"
Jul 27 18:33:09.408895 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.408816    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f80c1d4-c0f2-4362-890e-36d864925a76" containerName="console"
Jul 27 18:33:09.408895 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.408871    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f80c1d4-c0f2-4362-890e-36d864925a76" containerName="console"
Jul 27 18:33:09.412727 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.412706    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:33:09.424164 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.424140    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58cf7c967-q62dt"]
Jul 27 18:33:09.500002 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.499971    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-console-config\") pod \"console-58cf7c967-q62dt\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") " pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:33:09.500207 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.500010    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pvvs\" (UniqueName: \"kubernetes.io/projected/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-kube-api-access-6pvvs\") pod \"console-58cf7c967-q62dt\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") " pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:33:09.500207 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.500040    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-oauth-serving-cert\") pod \"console-58cf7c967-q62dt\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") " pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:33:09.500207 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.500057    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-console-serving-cert\") pod \"console-58cf7c967-q62dt\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") " pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:33:09.500207 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.500113    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-console-oauth-config\") pod \"console-58cf7c967-q62dt\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") " pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:33:09.500207 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.500154    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-service-ca\") pod \"console-58cf7c967-q62dt\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") " pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:33:09.500207 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.500177    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-trusted-ca-bundle\") pod \"console-58cf7c967-q62dt\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") " pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:33:09.601065 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.601024    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-console-config\") pod \"console-58cf7c967-q62dt\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") " pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:33:09.601065 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.601066    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pvvs\" (UniqueName: \"kubernetes.io/projected/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-kube-api-access-6pvvs\") pod \"console-58cf7c967-q62dt\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") " pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:33:09.601317 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.601201    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-oauth-serving-cert\") pod \"console-58cf7c967-q62dt\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") " pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:33:09.601317 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.601248    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-console-serving-cert\") pod \"console-58cf7c967-q62dt\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") " pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:33:09.601317 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.601288    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-console-oauth-config\") pod \"console-58cf7c967-q62dt\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") " pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:33:09.601451 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.601318    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-service-ca\") pod \"console-58cf7c967-q62dt\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") " pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:33:09.601451 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.601356    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-trusted-ca-bundle\") pod \"console-58cf7c967-q62dt\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") " pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:33:09.601933 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.601907    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-oauth-serving-cert\") pod \"console-58cf7c967-q62dt\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") " pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:33:09.601933 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.601934    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-console-config\") pod \"console-58cf7c967-q62dt\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") " pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:33:09.602334 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.602216    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-trusted-ca-bundle\") pod \"console-58cf7c967-q62dt\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") " pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:33:09.602334 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.602252    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-service-ca\") pod \"console-58cf7c967-q62dt\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") " pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:33:09.603863 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.603831    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-console-serving-cert\") pod \"console-58cf7c967-q62dt\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") " pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:33:09.603983 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.603965    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-console-oauth-config\") pod \"console-58cf7c967-q62dt\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") " pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:33:09.608918 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.608900    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pvvs\" (UniqueName: \"kubernetes.io/projected/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-kube-api-access-6pvvs\") pod \"console-58cf7c967-q62dt\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") " pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:33:09.721245 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.721154    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:33:09.852415 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:09.852380    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58cf7c967-q62dt"]
Jul 27 18:33:09.855183 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:33:09.855157    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3023d8c_9812_4e80_93a4_e8f4ee0ab736.slice/crio-104054455f8c152e4b5e3cc3a71613247a8a0e648ba5e048b1e2cae21b7eb716 WatchSource:0}: Error finding container 104054455f8c152e4b5e3cc3a71613247a8a0e648ba5e048b1e2cae21b7eb716: Status 404 returned error can't find the container with id 104054455f8c152e4b5e3cc3a71613247a8a0e648ba5e048b1e2cae21b7eb716
Jul 27 18:33:10.268716 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:10.268682    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58cf7c967-q62dt" event={"ID":"b3023d8c-9812-4e80-93a4-e8f4ee0ab736","Type":"ContainerStarted","Data":"b22c70b13e70b9df05484db97e1c6983fe7b42f49a07d6cf539a9ce4a6ad4383"}
Jul 27 18:33:10.268910 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:10.268730    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58cf7c967-q62dt" event={"ID":"b3023d8c-9812-4e80-93a4-e8f4ee0ab736","Type":"ContainerStarted","Data":"104054455f8c152e4b5e3cc3a71613247a8a0e648ba5e048b1e2cae21b7eb716"}
Jul 27 18:33:10.288505 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:10.288456    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58cf7c967-q62dt" podStartSLOduration=1.288439764 podStartE2EDuration="1.288439764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:33:10.287308798 +0000 UTC m=+250.721566412" watchObservedRunningTime="2026-07-27 18:33:10.288439764 +0000 UTC m=+250.722697375"
Jul 27 18:33:19.722238 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:19.722187    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:33:19.722238 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:19.722258    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:33:19.726961 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:19.726936    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:33:20.312005 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:20.311931    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:33:20.356494 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:20.356461    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c487c5dc9-2g97l"]
Jul 27 18:33:45.379247 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:45.379183    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c487c5dc9-2g97l" podUID="73ba6f74-1b4d-4427-a1a6-29de11bfd2f4" containerName="console" containerID="cri-o://30181192308cc49028547297cb21bf4e348bb3463b20004ee93984c57891cb66" gracePeriod=15
Jul 27 18:33:45.624311 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:45.624289    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:33:45.696414 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:45.696333    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-console-oauth-config\") pod \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") "
Jul 27 18:33:45.696414 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:45.696381    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-console-serving-cert\") pod \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") "
Jul 27 18:33:45.696414 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:45.696408    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-console-config\") pod \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") "
Jul 27 18:33:45.696674 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:45.696440    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-oauth-serving-cert\") pod \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") "
Jul 27 18:33:45.696674 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:45.696466    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzlc4\" (UniqueName: \"kubernetes.io/projected/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-kube-api-access-lzlc4\") pod \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") "
Jul 27 18:33:45.696674 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:45.696492    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-trusted-ca-bundle\") pod \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") "
Jul 27 18:33:45.696674 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:45.696531    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-service-ca\") pod \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\" (UID: \"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4\") "
Jul 27 18:33:45.696994 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:45.696899    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-console-config" (OuterVolumeSpecName: "console-config") pod "73ba6f74-1b4d-4427-a1a6-29de11bfd2f4" (UID: "73ba6f74-1b4d-4427-a1a6-29de11bfd2f4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:33:45.697147 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:45.697024    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-service-ca" (OuterVolumeSpecName: "service-ca") pod "73ba6f74-1b4d-4427-a1a6-29de11bfd2f4" (UID: "73ba6f74-1b4d-4427-a1a6-29de11bfd2f4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:33:45.697147 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:45.697076    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "73ba6f74-1b4d-4427-a1a6-29de11bfd2f4" (UID: "73ba6f74-1b4d-4427-a1a6-29de11bfd2f4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:33:45.697147 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:45.697099    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "73ba6f74-1b4d-4427-a1a6-29de11bfd2f4" (UID: "73ba6f74-1b4d-4427-a1a6-29de11bfd2f4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:33:45.698717 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:45.698690    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "73ba6f74-1b4d-4427-a1a6-29de11bfd2f4" (UID: "73ba6f74-1b4d-4427-a1a6-29de11bfd2f4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:33:45.698821 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:45.698780    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "73ba6f74-1b4d-4427-a1a6-29de11bfd2f4" (UID: "73ba6f74-1b4d-4427-a1a6-29de11bfd2f4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:33:45.699055 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:45.699021    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-kube-api-access-lzlc4" (OuterVolumeSpecName: "kube-api-access-lzlc4") pod "73ba6f74-1b4d-4427-a1a6-29de11bfd2f4" (UID: "73ba6f74-1b4d-4427-a1a6-29de11bfd2f4"). InnerVolumeSpecName "kube-api-access-lzlc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:33:45.797140 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:45.797079    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-console-oauth-config\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:33:45.797140 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:45.797109    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-console-serving-cert\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:33:45.797140 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:45.797144    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-console-config\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:33:45.797369 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:45.797160    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-oauth-serving-cert\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:33:45.797369 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:45.797175    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzlc4\" (UniqueName: \"kubernetes.io/projected/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-kube-api-access-lzlc4\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:33:45.797369 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:45.797187    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-trusted-ca-bundle\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:33:45.797369 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:45.797196    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4-service-ca\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:33:46.406982 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:46.406898    2566 generic.go:356] "Generic (PLEG): container finished" podID="73ba6f74-1b4d-4427-a1a6-29de11bfd2f4" containerID="30181192308cc49028547297cb21bf4e348bb3463b20004ee93984c57891cb66" exitCode=0
Jul 27 18:33:46.406982 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:46.406953    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c487c5dc9-2g97l"
Jul 27 18:33:46.407483 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:46.406985    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c487c5dc9-2g97l" event={"ID":"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4","Type":"ContainerDied","Data":"30181192308cc49028547297cb21bf4e348bb3463b20004ee93984c57891cb66"}
Jul 27 18:33:46.407483 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:46.407043    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c487c5dc9-2g97l" event={"ID":"73ba6f74-1b4d-4427-a1a6-29de11bfd2f4","Type":"ContainerDied","Data":"7128d92c780c90b6b97e9005ab8466457cfc409275f8555b9b6a9006543f0744"}
Jul 27 18:33:46.407483 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:46.407075    2566 scope.go:117] "RemoveContainer" containerID="30181192308cc49028547297cb21bf4e348bb3463b20004ee93984c57891cb66"
Jul 27 18:33:46.415423 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:46.415407    2566 scope.go:117] "RemoveContainer" containerID="30181192308cc49028547297cb21bf4e348bb3463b20004ee93984c57891cb66"
Jul 27 18:33:46.415665 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:33:46.415646    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30181192308cc49028547297cb21bf4e348bb3463b20004ee93984c57891cb66\": container with ID starting with 30181192308cc49028547297cb21bf4e348bb3463b20004ee93984c57891cb66 not found: ID does not exist" containerID="30181192308cc49028547297cb21bf4e348bb3463b20004ee93984c57891cb66"
Jul 27 18:33:46.415728 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:46.415672    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30181192308cc49028547297cb21bf4e348bb3463b20004ee93984c57891cb66"} err="failed to get container status \"30181192308cc49028547297cb21bf4e348bb3463b20004ee93984c57891cb66\": rpc error: code = NotFound desc = could not find container \"30181192308cc49028547297cb21bf4e348bb3463b20004ee93984c57891cb66\": container with ID starting with 30181192308cc49028547297cb21bf4e348bb3463b20004ee93984c57891cb66 not found: ID does not exist"
Jul 27 18:33:46.423543 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:46.423518    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c487c5dc9-2g97l"]
Jul 27 18:33:46.428813 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:46.428795    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c487c5dc9-2g97l"]
Jul 27 18:33:48.123026 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:48.122996    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73ba6f74-1b4d-4427-a1a6-29de11bfd2f4" path="/var/lib/kubelet/pods/73ba6f74-1b4d-4427-a1a6-29de11bfd2f4/volumes"
Jul 27 18:33:59.958660 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:59.958628    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 18:33:59.966234 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:59.966209    2566 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 18:33:59.970367 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:33:59.970346    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 18:34:07.953273 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:07.953238    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d577d56cb-g9nlf"]
Jul 27 18:34:07.953660 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:07.953541    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73ba6f74-1b4d-4427-a1a6-29de11bfd2f4" containerName="console"
Jul 27 18:34:07.953660 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:07.953552    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="73ba6f74-1b4d-4427-a1a6-29de11bfd2f4" containerName="console"
Jul 27 18:34:07.953660 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:07.953612    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="73ba6f74-1b4d-4427-a1a6-29de11bfd2f4" containerName="console"
Jul 27 18:34:07.955472 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:07.955455    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d577d56cb-g9nlf"
Jul 27 18:34:07.960023 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:07.959971    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-zccds\""
Jul 27 18:34:07.960182 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:07.960138    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 27 18:34:07.960248 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:07.960186    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 27 18:34:07.960248 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:07.960186    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 27 18:34:07.960248 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:07.960227    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 27 18:34:07.965194 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:07.965170    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d577d56cb-g9nlf"]
Jul 27 18:34:08.043827 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:08.043796    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd86cf849-g8ndv"]
Jul 27 18:34:08.046204 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:08.046184    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd86cf849-g8ndv"
Jul 27 18:34:08.048878 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:08.048858    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 27 18:34:08.058479 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:08.058453    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd86cf849-g8ndv"]
Jul 27 18:34:08.083606 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:08.083570    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ea2177ed-5234-4564-b307-6088bcd145cb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d577d56cb-g9nlf\" (UID: \"ea2177ed-5234-4564-b307-6088bcd145cb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d577d56cb-g9nlf"
Jul 27 18:34:08.083769 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:08.083616    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlstr\" (UniqueName: \"kubernetes.io/projected/ea2177ed-5234-4564-b307-6088bcd145cb-kube-api-access-xlstr\") pod \"managed-serviceaccount-addon-agent-7d577d56cb-g9nlf\" (UID: \"ea2177ed-5234-4564-b307-6088bcd145cb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d577d56cb-g9nlf"
Jul 27 18:34:08.184253 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:08.184203    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dab2006f-da67-4ec8-bfe7-10c6d9da9c79-tmp\") pod \"klusterlet-addon-workmgr-7cd86cf849-g8ndv\" (UID: \"dab2006f-da67-4ec8-bfe7-10c6d9da9c79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd86cf849-g8ndv"
Jul 27 18:34:08.184449 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:08.184288    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66gdn\" (UniqueName: \"kubernetes.io/projected/dab2006f-da67-4ec8-bfe7-10c6d9da9c79-kube-api-access-66gdn\") pod \"klusterlet-addon-workmgr-7cd86cf849-g8ndv\" (UID: \"dab2006f-da67-4ec8-bfe7-10c6d9da9c79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd86cf849-g8ndv"
Jul 27 18:34:08.184449 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:08.184312    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/dab2006f-da67-4ec8-bfe7-10c6d9da9c79-klusterlet-config\") pod \"klusterlet-addon-workmgr-7cd86cf849-g8ndv\" (UID: \"dab2006f-da67-4ec8-bfe7-10c6d9da9c79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd86cf849-g8ndv"
Jul 27 18:34:08.184449 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:08.184343    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ea2177ed-5234-4564-b307-6088bcd145cb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d577d56cb-g9nlf\" (UID: \"ea2177ed-5234-4564-b307-6088bcd145cb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d577d56cb-g9nlf"
Jul 27 18:34:08.184449 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:08.184388    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlstr\" (UniqueName: \"kubernetes.io/projected/ea2177ed-5234-4564-b307-6088bcd145cb-kube-api-access-xlstr\") pod \"managed-serviceaccount-addon-agent-7d577d56cb-g9nlf\" (UID: \"ea2177ed-5234-4564-b307-6088bcd145cb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d577d56cb-g9nlf"
Jul 27 18:34:08.186926 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:08.186899    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ea2177ed-5234-4564-b307-6088bcd145cb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d577d56cb-g9nlf\" (UID: \"ea2177ed-5234-4564-b307-6088bcd145cb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d577d56cb-g9nlf"
Jul 27 18:34:08.193018 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:08.192997    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlstr\" (UniqueName: \"kubernetes.io/projected/ea2177ed-5234-4564-b307-6088bcd145cb-kube-api-access-xlstr\") pod \"managed-serviceaccount-addon-agent-7d577d56cb-g9nlf\" (UID: \"ea2177ed-5234-4564-b307-6088bcd145cb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d577d56cb-g9nlf"
Jul 27 18:34:08.273664 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:08.273628    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d577d56cb-g9nlf"
Jul 27 18:34:08.285111 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:08.285089    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66gdn\" (UniqueName: \"kubernetes.io/projected/dab2006f-da67-4ec8-bfe7-10c6d9da9c79-kube-api-access-66gdn\") pod \"klusterlet-addon-workmgr-7cd86cf849-g8ndv\" (UID: \"dab2006f-da67-4ec8-bfe7-10c6d9da9c79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd86cf849-g8ndv"
Jul 27 18:34:08.285213 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:08.285147    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/dab2006f-da67-4ec8-bfe7-10c6d9da9c79-klusterlet-config\") pod \"klusterlet-addon-workmgr-7cd86cf849-g8ndv\" (UID: \"dab2006f-da67-4ec8-bfe7-10c6d9da9c79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd86cf849-g8ndv"
Jul 27 18:34:08.285274 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:08.285232    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dab2006f-da67-4ec8-bfe7-10c6d9da9c79-tmp\") pod \"klusterlet-addon-workmgr-7cd86cf849-g8ndv\" (UID: \"dab2006f-da67-4ec8-bfe7-10c6d9da9c79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd86cf849-g8ndv"
Jul 27 18:34:08.285629 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:08.285605    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dab2006f-da67-4ec8-bfe7-10c6d9da9c79-tmp\") pod \"klusterlet-addon-workmgr-7cd86cf849-g8ndv\" (UID: \"dab2006f-da67-4ec8-bfe7-10c6d9da9c79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd86cf849-g8ndv"
Jul 27 18:34:08.288137 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:08.288094    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/dab2006f-da67-4ec8-bfe7-10c6d9da9c79-klusterlet-config\") pod \"klusterlet-addon-workmgr-7cd86cf849-g8ndv\" (UID: \"dab2006f-da67-4ec8-bfe7-10c6d9da9c79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd86cf849-g8ndv"
Jul 27 18:34:08.295004 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:08.294984    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66gdn\" (UniqueName: \"kubernetes.io/projected/dab2006f-da67-4ec8-bfe7-10c6d9da9c79-kube-api-access-66gdn\") pod \"klusterlet-addon-workmgr-7cd86cf849-g8ndv\" (UID: \"dab2006f-da67-4ec8-bfe7-10c6d9da9c79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd86cf849-g8ndv"
Jul 27 18:34:08.361535 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:08.361505    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd86cf849-g8ndv"
Jul 27 18:34:08.404465 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:08.404434    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d577d56cb-g9nlf"]
Jul 27 18:34:08.406386 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:34:08.406362    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea2177ed_5234_4564_b307_6088bcd145cb.slice/crio-3eb0610924202cd1a0dec06852ff1f4922d126e1d49d4ef607b5f9be39c8399b WatchSource:0}: Error finding container 3eb0610924202cd1a0dec06852ff1f4922d126e1d49d4ef607b5f9be39c8399b: Status 404 returned error can't find the container with id 3eb0610924202cd1a0dec06852ff1f4922d126e1d49d4ef607b5f9be39c8399b
Jul 27 18:34:08.408423 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:08.408406    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 18:34:08.491354 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:08.491320    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d577d56cb-g9nlf" event={"ID":"ea2177ed-5234-4564-b307-6088bcd145cb","Type":"ContainerStarted","Data":"3eb0610924202cd1a0dec06852ff1f4922d126e1d49d4ef607b5f9be39c8399b"}
Jul 27 18:34:08.494073 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:08.494046    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd86cf849-g8ndv"]
Jul 27 18:34:08.497083 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:34:08.497057    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab2006f_da67_4ec8_bfe7_10c6d9da9c79.slice/crio-02b77eb46451428c833ce370f1f5220458eb87b21c33c44fd188d4866c11040a WatchSource:0}: Error finding container 02b77eb46451428c833ce370f1f5220458eb87b21c33c44fd188d4866c11040a: Status 404 returned error can't find the container with id 02b77eb46451428c833ce370f1f5220458eb87b21c33c44fd188d4866c11040a
Jul 27 18:34:09.499868 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:09.499781    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd86cf849-g8ndv" event={"ID":"dab2006f-da67-4ec8-bfe7-10c6d9da9c79","Type":"ContainerStarted","Data":"02b77eb46451428c833ce370f1f5220458eb87b21c33c44fd188d4866c11040a"}
Jul 27 18:34:12.514505 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:12.514402    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d577d56cb-g9nlf" event={"ID":"ea2177ed-5234-4564-b307-6088bcd145cb","Type":"ContainerStarted","Data":"2c42d98a41edea6ac491d1a7df9dea669c74511c69f523b3b02550e99b14a44a"}
Jul 27 18:34:12.515894 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:12.515867    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd86cf849-g8ndv" event={"ID":"dab2006f-da67-4ec8-bfe7-10c6d9da9c79","Type":"ContainerStarted","Data":"ed09e1b60767dd1e4f2bf8a057f8a0bc1efadb27c52b5a9e70cc36bb9305da88"}
Jul 27 18:34:12.516072 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:12.516053    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd86cf849-g8ndv"
Jul 27 18:34:12.517816 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:12.517800    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd86cf849-g8ndv"
Jul 27 18:34:12.533092 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:12.533042    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d577d56cb-g9nlf" podStartSLOduration=1.730920039 podStartE2EDuration="5.53302977s" totalImagesPullingTime="3.802109731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:34:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:34:12.531818948 +0000 UTC m=+312.966076559" watchObservedRunningTime="2026-07-27 18:34:12.53302977 +0000 UTC m=+312.967287444"
Jul 27 18:34:12.553812 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:12.553766    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7cd86cf849-g8ndv" podStartSLOduration=0.831648623 podStartE2EDuration="4.553751949s" totalImagesPullingTime="3.722103326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:34:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:34:12.552581493 +0000 UTC m=+312.986839106" watchObservedRunningTime="2026-07-27 18:34:12.553751949 +0000 UTC m=+312.988009561"
Jul 27 18:34:24.436096 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:24.436062    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6"]
Jul 27 18:34:24.439443 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:24.439427    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6"
Jul 27 18:34:24.442713 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:24.442684    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 27 18:34:24.442827 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:24.442781    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 27 18:34:24.443851 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:24.443838    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-s2xv2\""
Jul 27 18:34:24.449180 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:24.449132    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6"]
Jul 27 18:34:24.526908 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:24.526867    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b86xn\" (UniqueName: \"kubernetes.io/projected/3593fa19-3c7f-4d4c-9aa5-99f81f978215-kube-api-access-b86xn\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6\" (UID: \"3593fa19-3c7f-4d4c-9aa5-99f81f978215\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6"
Jul 27 18:34:24.527092 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:24.526915    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3593fa19-3c7f-4d4c-9aa5-99f81f978215-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6\" (UID: \"3593fa19-3c7f-4d4c-9aa5-99f81f978215\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6"
Jul 27 18:34:24.527092 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:24.527022    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3593fa19-3c7f-4d4c-9aa5-99f81f978215-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6\" (UID: \"3593fa19-3c7f-4d4c-9aa5-99f81f978215\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6"
Jul 27 18:34:24.627770 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:24.627734    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3593fa19-3c7f-4d4c-9aa5-99f81f978215-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6\" (UID: \"3593fa19-3c7f-4d4c-9aa5-99f81f978215\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6"
Jul 27 18:34:24.627933 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:24.627781    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b86xn\" (UniqueName: \"kubernetes.io/projected/3593fa19-3c7f-4d4c-9aa5-99f81f978215-kube-api-access-b86xn\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6\" (UID: \"3593fa19-3c7f-4d4c-9aa5-99f81f978215\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6"
Jul 27 18:34:24.627933 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:24.627801    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3593fa19-3c7f-4d4c-9aa5-99f81f978215-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6\" (UID: \"3593fa19-3c7f-4d4c-9aa5-99f81f978215\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6"
Jul 27 18:34:24.628158 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:24.628111    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3593fa19-3c7f-4d4c-9aa5-99f81f978215-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6\" (UID: \"3593fa19-3c7f-4d4c-9aa5-99f81f978215\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6"
Jul 27 18:34:24.628158 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:24.628151    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3593fa19-3c7f-4d4c-9aa5-99f81f978215-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6\" (UID: \"3593fa19-3c7f-4d4c-9aa5-99f81f978215\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6"
Jul 27 18:34:24.638255 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:24.638226    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b86xn\" (UniqueName: \"kubernetes.io/projected/3593fa19-3c7f-4d4c-9aa5-99f81f978215-kube-api-access-b86xn\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6\" (UID: \"3593fa19-3c7f-4d4c-9aa5-99f81f978215\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6"
Jul 27 18:34:24.749083 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:24.748985    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6"
Jul 27 18:34:24.879107 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:24.879069    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6"]
Jul 27 18:34:24.881858 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:34:24.881830    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3593fa19_3c7f_4d4c_9aa5_99f81f978215.slice/crio-a02dcc9edc854688bdeafb94bcd3af836d0a3f046f4dec916bf6e93922eb9eee WatchSource:0}: Error finding container a02dcc9edc854688bdeafb94bcd3af836d0a3f046f4dec916bf6e93922eb9eee: Status 404 returned error can't find the container with id a02dcc9edc854688bdeafb94bcd3af836d0a3f046f4dec916bf6e93922eb9eee
Jul 27 18:34:25.568932 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:25.568898    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6" event={"ID":"3593fa19-3c7f-4d4c-9aa5-99f81f978215","Type":"ContainerStarted","Data":"a02dcc9edc854688bdeafb94bcd3af836d0a3f046f4dec916bf6e93922eb9eee"}
Jul 27 18:34:32.602304 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:32.602261    2566 generic.go:356] "Generic (PLEG): container finished" podID="3593fa19-3c7f-4d4c-9aa5-99f81f978215" containerID="43be0981700bcc8532a7a618a069aa3852fc05c0c7c327a64f0adf9f09c7a9ef" exitCode=0
Jul 27 18:34:32.602699 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:32.602335    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6" event={"ID":"3593fa19-3c7f-4d4c-9aa5-99f81f978215","Type":"ContainerDied","Data":"43be0981700bcc8532a7a618a069aa3852fc05c0c7c327a64f0adf9f09c7a9ef"}
Jul 27 18:34:35.616970 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:35.616931    2566 generic.go:356] "Generic (PLEG): container finished" podID="3593fa19-3c7f-4d4c-9aa5-99f81f978215" containerID="4f21cdebfad47277890686fcc7290a4ef5069e9926c4675f879ea3bbfa71e46f" exitCode=0
Jul 27 18:34:35.617421 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:35.617019    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6" event={"ID":"3593fa19-3c7f-4d4c-9aa5-99f81f978215","Type":"ContainerDied","Data":"4f21cdebfad47277890686fcc7290a4ef5069e9926c4675f879ea3bbfa71e46f"}
Jul 27 18:34:44.657831 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:44.657796    2566 generic.go:356] "Generic (PLEG): container finished" podID="3593fa19-3c7f-4d4c-9aa5-99f81f978215" containerID="03fcae14d10a09ab9f9037be6193ffd30fcaddba851dbc196397491200613d4f" exitCode=0
Jul 27 18:34:44.658252 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:44.657882    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6" event={"ID":"3593fa19-3c7f-4d4c-9aa5-99f81f978215","Type":"ContainerDied","Data":"03fcae14d10a09ab9f9037be6193ffd30fcaddba851dbc196397491200613d4f"}
Jul 27 18:34:45.796608 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:45.796585    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6"
Jul 27 18:34:45.909992 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:45.909959    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3593fa19-3c7f-4d4c-9aa5-99f81f978215-bundle\") pod \"3593fa19-3c7f-4d4c-9aa5-99f81f978215\" (UID: \"3593fa19-3c7f-4d4c-9aa5-99f81f978215\") "
Jul 27 18:34:45.910215 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:45.910050    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3593fa19-3c7f-4d4c-9aa5-99f81f978215-util\") pod \"3593fa19-3c7f-4d4c-9aa5-99f81f978215\" (UID: \"3593fa19-3c7f-4d4c-9aa5-99f81f978215\") "
Jul 27 18:34:45.910215 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:45.910077    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b86xn\" (UniqueName: \"kubernetes.io/projected/3593fa19-3c7f-4d4c-9aa5-99f81f978215-kube-api-access-b86xn\") pod \"3593fa19-3c7f-4d4c-9aa5-99f81f978215\" (UID: \"3593fa19-3c7f-4d4c-9aa5-99f81f978215\") "
Jul 27 18:34:45.910700 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:45.910671    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3593fa19-3c7f-4d4c-9aa5-99f81f978215-bundle" (OuterVolumeSpecName: "bundle") pod "3593fa19-3c7f-4d4c-9aa5-99f81f978215" (UID: "3593fa19-3c7f-4d4c-9aa5-99f81f978215"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:34:45.912389 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:45.912369    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3593fa19-3c7f-4d4c-9aa5-99f81f978215-kube-api-access-b86xn" (OuterVolumeSpecName: "kube-api-access-b86xn") pod "3593fa19-3c7f-4d4c-9aa5-99f81f978215" (UID: "3593fa19-3c7f-4d4c-9aa5-99f81f978215"). InnerVolumeSpecName "kube-api-access-b86xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:34:45.917253 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:45.917205    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3593fa19-3c7f-4d4c-9aa5-99f81f978215-util" (OuterVolumeSpecName: "util") pod "3593fa19-3c7f-4d4c-9aa5-99f81f978215" (UID: "3593fa19-3c7f-4d4c-9aa5-99f81f978215"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:34:46.011165 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:46.011107    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3593fa19-3c7f-4d4c-9aa5-99f81f978215-bundle\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:34:46.011165 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:46.011162    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3593fa19-3c7f-4d4c-9aa5-99f81f978215-util\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:34:46.011165 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:46.011171    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b86xn\" (UniqueName: \"kubernetes.io/projected/3593fa19-3c7f-4d4c-9aa5-99f81f978215-kube-api-access-b86xn\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:34:46.667464 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:46.667431    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6" event={"ID":"3593fa19-3c7f-4d4c-9aa5-99f81f978215","Type":"ContainerDied","Data":"a02dcc9edc854688bdeafb94bcd3af836d0a3f046f4dec916bf6e93922eb9eee"}
Jul 27 18:34:46.667464 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:46.667449    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nxnf6"
Jul 27 18:34:46.667665 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:46.667481    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a02dcc9edc854688bdeafb94bcd3af836d0a3f046f4dec916bf6e93922eb9eee"
Jul 27 18:34:52.482615 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:52.482579    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8nt7w"]
Jul 27 18:34:52.483050 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:52.482894    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3593fa19-3c7f-4d4c-9aa5-99f81f978215" containerName="pull"
Jul 27 18:34:52.483050 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:52.482905    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3593fa19-3c7f-4d4c-9aa5-99f81f978215" containerName="pull"
Jul 27 18:34:52.483050 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:52.482917    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3593fa19-3c7f-4d4c-9aa5-99f81f978215" containerName="extract"
Jul 27 18:34:52.483050 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:52.482923    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3593fa19-3c7f-4d4c-9aa5-99f81f978215" containerName="extract"
Jul 27 18:34:52.483050 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:52.482933    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3593fa19-3c7f-4d4c-9aa5-99f81f978215" containerName="util"
Jul 27 18:34:52.483050 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:52.482938    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3593fa19-3c7f-4d4c-9aa5-99f81f978215" containerName="util"
Jul 27 18:34:52.483050 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:52.482994    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3593fa19-3c7f-4d4c-9aa5-99f81f978215" containerName="extract"
Jul 27 18:34:52.484852 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:52.484837    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8nt7w"
Jul 27 18:34:52.488937 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:52.488913    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 27 18:34:52.489043 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:52.488976    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-rjrlp\""
Jul 27 18:34:52.489326 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:52.489312    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 27 18:34:52.508515 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:52.508494    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8nt7w"]
Jul 27 18:34:52.571638 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:52.571605    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b774b754-7251-4b8e-a40a-faa9371b62f4-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-8nt7w\" (UID: \"b774b754-7251-4b8e-a40a-faa9371b62f4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8nt7w"
Jul 27 18:34:52.571798 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:52.571643    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flhbl\" (UniqueName: \"kubernetes.io/projected/b774b754-7251-4b8e-a40a-faa9371b62f4-kube-api-access-flhbl\") pod \"cert-manager-operator-controller-manager-55c444bf86-8nt7w\" (UID: \"b774b754-7251-4b8e-a40a-faa9371b62f4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8nt7w"
Jul 27 18:34:52.672480 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:52.672435    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b774b754-7251-4b8e-a40a-faa9371b62f4-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-8nt7w\" (UID: \"b774b754-7251-4b8e-a40a-faa9371b62f4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8nt7w"
Jul 27 18:34:52.672480 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:52.672482    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flhbl\" (UniqueName: \"kubernetes.io/projected/b774b754-7251-4b8e-a40a-faa9371b62f4-kube-api-access-flhbl\") pod \"cert-manager-operator-controller-manager-55c444bf86-8nt7w\" (UID: \"b774b754-7251-4b8e-a40a-faa9371b62f4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8nt7w"
Jul 27 18:34:52.672816 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:52.672798    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b774b754-7251-4b8e-a40a-faa9371b62f4-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-8nt7w\" (UID: \"b774b754-7251-4b8e-a40a-faa9371b62f4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8nt7w"
Jul 27 18:34:52.682110 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:52.682082    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flhbl\" (UniqueName: \"kubernetes.io/projected/b774b754-7251-4b8e-a40a-faa9371b62f4-kube-api-access-flhbl\") pod \"cert-manager-operator-controller-manager-55c444bf86-8nt7w\" (UID: \"b774b754-7251-4b8e-a40a-faa9371b62f4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8nt7w"
Jul 27 18:34:52.793514 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:52.793479    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8nt7w"
Jul 27 18:34:52.931370 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:52.931343    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8nt7w"]
Jul 27 18:34:52.935093 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:34:52.935061    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb774b754_7251_4b8e_a40a_faa9371b62f4.slice/crio-4ae77338b0bd315bf9d0340ad9c384f98365f493fd3873bb61dce3785098d13b WatchSource:0}: Error finding container 4ae77338b0bd315bf9d0340ad9c384f98365f493fd3873bb61dce3785098d13b: Status 404 returned error can't find the container with id 4ae77338b0bd315bf9d0340ad9c384f98365f493fd3873bb61dce3785098d13b
Jul 27 18:34:53.699804 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:53.699771    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8nt7w" event={"ID":"b774b754-7251-4b8e-a40a-faa9371b62f4","Type":"ContainerStarted","Data":"4ae77338b0bd315bf9d0340ad9c384f98365f493fd3873bb61dce3785098d13b"}
Jul 27 18:34:54.707540 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:54.707506    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8nt7w" event={"ID":"b774b754-7251-4b8e-a40a-faa9371b62f4","Type":"ContainerStarted","Data":"d79dfcb39daa0d420be502a624d6c814d59f279a9aaf283851118cd508221ba1"}
Jul 27 18:34:54.754605 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:54.754549    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8nt7w" podStartSLOduration=1.11890229 podStartE2EDuration="2.754533213s" totalImagesPullingTime="1.635630923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:34:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:34:54.75422297 +0000 UTC m=+355.188480581" watchObservedRunningTime="2026-07-27 18:34:54.754533213 +0000 UTC m=+355.188790824"
Jul 27 18:34:55.850428 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:55.850391    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-fxvzg"]
Jul 27 18:34:55.854018 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:55.853997    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-fxvzg"
Jul 27 18:34:55.856902 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:55.856885    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 27 18:34:55.857217 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:55.857199    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 27 18:34:55.858194 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:55.858178    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-5jbgv\""
Jul 27 18:34:55.864837 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:55.864792    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-fxvzg"]
Jul 27 18:34:56.002597 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:56.002540    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn4j4\" (UniqueName: \"kubernetes.io/projected/5103679a-8d35-4f00-8efc-11eff0351a17-kube-api-access-qn4j4\") pod \"cert-manager-webhook-6486ffd997-fxvzg\" (UID: \"5103679a-8d35-4f00-8efc-11eff0351a17\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fxvzg"
Jul 27 18:34:56.002776 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:56.002658    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5103679a-8d35-4f00-8efc-11eff0351a17-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-fxvzg\" (UID: \"5103679a-8d35-4f00-8efc-11eff0351a17\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fxvzg"
Jul 27 18:34:56.103883 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:56.103800    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5103679a-8d35-4f00-8efc-11eff0351a17-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-fxvzg\" (UID: \"5103679a-8d35-4f00-8efc-11eff0351a17\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fxvzg"
Jul 27 18:34:56.103883 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:56.103862    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn4j4\" (UniqueName: \"kubernetes.io/projected/5103679a-8d35-4f00-8efc-11eff0351a17-kube-api-access-qn4j4\") pod \"cert-manager-webhook-6486ffd997-fxvzg\" (UID: \"5103679a-8d35-4f00-8efc-11eff0351a17\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fxvzg"
Jul 27 18:34:56.115678 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:56.115645    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5103679a-8d35-4f00-8efc-11eff0351a17-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-fxvzg\" (UID: \"5103679a-8d35-4f00-8efc-11eff0351a17\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fxvzg"
Jul 27 18:34:56.115927 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:56.115904    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn4j4\" (UniqueName: \"kubernetes.io/projected/5103679a-8d35-4f00-8efc-11eff0351a17-kube-api-access-qn4j4\") pod \"cert-manager-webhook-6486ffd997-fxvzg\" (UID: \"5103679a-8d35-4f00-8efc-11eff0351a17\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fxvzg"
Jul 27 18:34:56.170834 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:56.170794    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-fxvzg"
Jul 27 18:34:56.231188 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:56.230862    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn"]
Jul 27 18:34:56.237976 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:56.237953    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn"
Jul 27 18:34:56.241937 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:56.241889    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 27 18:34:56.242196 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:56.242177    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-s2xv2\""
Jul 27 18:34:56.242297 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:56.242198    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 27 18:34:56.247377 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:56.247344    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn"]
Jul 27 18:34:56.311494 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:56.311471    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-fxvzg"]
Jul 27 18:34:56.313689 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:34:56.313667    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5103679a_8d35_4f00_8efc_11eff0351a17.slice/crio-97cd0fb0ef9db20a9910912df5454ee9071f2ceab343b90dd34f1ac6e46c0519 WatchSource:0}: Error finding container 97cd0fb0ef9db20a9910912df5454ee9071f2ceab343b90dd34f1ac6e46c0519: Status 404 returned error can't find the container with id 97cd0fb0ef9db20a9910912df5454ee9071f2ceab343b90dd34f1ac6e46c0519
Jul 27 18:34:56.407064 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:56.406972    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/065a521c-528c-4296-8e2c-8dcccd333eb7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn\" (UID: \"065a521c-528c-4296-8e2c-8dcccd333eb7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn"
Jul 27 18:34:56.407064 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:56.407026    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-847mg\" (UniqueName: \"kubernetes.io/projected/065a521c-528c-4296-8e2c-8dcccd333eb7-kube-api-access-847mg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn\" (UID: \"065a521c-528c-4296-8e2c-8dcccd333eb7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn"
Jul 27 18:34:56.407276 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:56.407145    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/065a521c-528c-4296-8e2c-8dcccd333eb7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn\" (UID: \"065a521c-528c-4296-8e2c-8dcccd333eb7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn"
Jul 27 18:34:56.508186 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:56.508155    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/065a521c-528c-4296-8e2c-8dcccd333eb7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn\" (UID: \"065a521c-528c-4296-8e2c-8dcccd333eb7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn"
Jul 27 18:34:56.508374 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:56.508217    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-847mg\" (UniqueName: \"kubernetes.io/projected/065a521c-528c-4296-8e2c-8dcccd333eb7-kube-api-access-847mg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn\" (UID: \"065a521c-528c-4296-8e2c-8dcccd333eb7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn"
Jul 27 18:34:56.508374 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:56.508277    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/065a521c-528c-4296-8e2c-8dcccd333eb7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn\" (UID: \"065a521c-528c-4296-8e2c-8dcccd333eb7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn"
Jul 27 18:34:56.508575 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:56.508554    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/065a521c-528c-4296-8e2c-8dcccd333eb7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn\" (UID: \"065a521c-528c-4296-8e2c-8dcccd333eb7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn"
Jul 27 18:34:56.508636 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:56.508586    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/065a521c-528c-4296-8e2c-8dcccd333eb7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn\" (UID: \"065a521c-528c-4296-8e2c-8dcccd333eb7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn"
Jul 27 18:34:56.517106 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:56.517084    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-847mg\" (UniqueName: \"kubernetes.io/projected/065a521c-528c-4296-8e2c-8dcccd333eb7-kube-api-access-847mg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn\" (UID: \"065a521c-528c-4296-8e2c-8dcccd333eb7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn"
Jul 27 18:34:56.549607 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:56.549583    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn"
Jul 27 18:34:56.684042 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:56.683859    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn"]
Jul 27 18:34:56.686646 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:34:56.686622    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065a521c_528c_4296_8e2c_8dcccd333eb7.slice/crio-a1d8a38ff1a9213a80ff8c2e0b9da5bcfb1e04a75a3c778e6199b744c6fe9803 WatchSource:0}: Error finding container a1d8a38ff1a9213a80ff8c2e0b9da5bcfb1e04a75a3c778e6199b744c6fe9803: Status 404 returned error can't find the container with id a1d8a38ff1a9213a80ff8c2e0b9da5bcfb1e04a75a3c778e6199b744c6fe9803
Jul 27 18:34:56.717506 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:56.717484    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn" event={"ID":"065a521c-528c-4296-8e2c-8dcccd333eb7","Type":"ContainerStarted","Data":"a1d8a38ff1a9213a80ff8c2e0b9da5bcfb1e04a75a3c778e6199b744c6fe9803"}
Jul 27 18:34:56.718856 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:56.718822    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-fxvzg" event={"ID":"5103679a-8d35-4f00-8efc-11eff0351a17","Type":"ContainerStarted","Data":"97cd0fb0ef9db20a9910912df5454ee9071f2ceab343b90dd34f1ac6e46c0519"}
Jul 27 18:34:57.726506 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:57.726470    2566 generic.go:356] "Generic (PLEG): container finished" podID="065a521c-528c-4296-8e2c-8dcccd333eb7" containerID="9e7a9f0fc6c2a993d5ca30a22f4c792e0637f30cc80384a298d2fd829dc8a96a" exitCode=0
Jul 27 18:34:57.726969 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:57.726561    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn" event={"ID":"065a521c-528c-4296-8e2c-8dcccd333eb7","Type":"ContainerDied","Data":"9e7a9f0fc6c2a993d5ca30a22f4c792e0637f30cc80384a298d2fd829dc8a96a"}
Jul 27 18:34:59.736760 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:59.736728    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-fxvzg" event={"ID":"5103679a-8d35-4f00-8efc-11eff0351a17","Type":"ContainerStarted","Data":"5b14294df984a63b1ecc259f41b48d92e4118f57f78f1bff5cfbe8a74a723241"}
Jul 27 18:34:59.737290 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:59.736788    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-fxvzg"
Jul 27 18:34:59.756968 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:34:59.756911    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-fxvzg" podStartSLOduration=2.016453555 podStartE2EDuration="4.75689583s" totalImagesPullingTime="2.740442275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:34:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:34:59.756438252 +0000 UTC m=+360.190695865" watchObservedRunningTime="2026-07-27 18:34:59.75689583 +0000 UTC m=+360.191153441"
Jul 27 18:35:00.743320 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:00.743217    2566 generic.go:356] "Generic (PLEG): container finished" podID="065a521c-528c-4296-8e2c-8dcccd333eb7" containerID="2aa9b0c34d5193c158ee9268c192caf52a4dceebb53861913b08aaa58ac12d22" exitCode=0
Jul 27 18:35:00.743320 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:00.743302    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn" event={"ID":"065a521c-528c-4296-8e2c-8dcccd333eb7","Type":"ContainerDied","Data":"2aa9b0c34d5193c158ee9268c192caf52a4dceebb53861913b08aaa58ac12d22"}
Jul 27 18:35:01.749814 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:01.749780    2566 generic.go:356] "Generic (PLEG): container finished" podID="065a521c-528c-4296-8e2c-8dcccd333eb7" containerID="932921a710df7333573104d3e805be9b8e9e15c4d1d97e0bb2eb8851dae26c5a" exitCode=0
Jul 27 18:35:01.749814 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:01.749819    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn" event={"ID":"065a521c-528c-4296-8e2c-8dcccd333eb7","Type":"ContainerDied","Data":"932921a710df7333573104d3e805be9b8e9e15c4d1d97e0bb2eb8851dae26c5a"}
Jul 27 18:35:02.902812 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:02.902780    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn"
Jul 27 18:35:03.065127 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:03.065098    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/065a521c-528c-4296-8e2c-8dcccd333eb7-util\") pod \"065a521c-528c-4296-8e2c-8dcccd333eb7\" (UID: \"065a521c-528c-4296-8e2c-8dcccd333eb7\") "
Jul 27 18:35:03.065277 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:03.065212    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/065a521c-528c-4296-8e2c-8dcccd333eb7-bundle\") pod \"065a521c-528c-4296-8e2c-8dcccd333eb7\" (UID: \"065a521c-528c-4296-8e2c-8dcccd333eb7\") "
Jul 27 18:35:03.065277 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:03.065255    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-847mg\" (UniqueName: \"kubernetes.io/projected/065a521c-528c-4296-8e2c-8dcccd333eb7-kube-api-access-847mg\") pod \"065a521c-528c-4296-8e2c-8dcccd333eb7\" (UID: \"065a521c-528c-4296-8e2c-8dcccd333eb7\") "
Jul 27 18:35:03.065594 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:03.065561    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065a521c-528c-4296-8e2c-8dcccd333eb7-bundle" (OuterVolumeSpecName: "bundle") pod "065a521c-528c-4296-8e2c-8dcccd333eb7" (UID: "065a521c-528c-4296-8e2c-8dcccd333eb7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:35:03.067449 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:03.067426    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065a521c-528c-4296-8e2c-8dcccd333eb7-kube-api-access-847mg" (OuterVolumeSpecName: "kube-api-access-847mg") pod "065a521c-528c-4296-8e2c-8dcccd333eb7" (UID: "065a521c-528c-4296-8e2c-8dcccd333eb7"). InnerVolumeSpecName "kube-api-access-847mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:35:03.071666 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:03.071642    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065a521c-528c-4296-8e2c-8dcccd333eb7-util" (OuterVolumeSpecName: "util") pod "065a521c-528c-4296-8e2c-8dcccd333eb7" (UID: "065a521c-528c-4296-8e2c-8dcccd333eb7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:35:03.166188 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:03.166143    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/065a521c-528c-4296-8e2c-8dcccd333eb7-bundle\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:35:03.166188 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:03.166182    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-847mg\" (UniqueName: \"kubernetes.io/projected/065a521c-528c-4296-8e2c-8dcccd333eb7-kube-api-access-847mg\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:35:03.166188 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:03.166192    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/065a521c-528c-4296-8e2c-8dcccd333eb7-util\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:35:03.761448 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:03.761417    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn" event={"ID":"065a521c-528c-4296-8e2c-8dcccd333eb7","Type":"ContainerDied","Data":"a1d8a38ff1a9213a80ff8c2e0b9da5bcfb1e04a75a3c778e6199b744c6fe9803"}
Jul 27 18:35:03.761593 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:03.761457    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6jsbn"
Jul 27 18:35:03.761593 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:03.761462    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1d8a38ff1a9213a80ff8c2e0b9da5bcfb1e04a75a3c778e6199b744c6fe9803"
Jul 27 18:35:05.745981 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:05.745946    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-fxvzg"
Jul 27 18:35:08.220265 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:08.220230    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lcncj"]
Jul 27 18:35:08.220648 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:08.220558    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="065a521c-528c-4296-8e2c-8dcccd333eb7" containerName="extract"
Jul 27 18:35:08.220648 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:08.220569    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="065a521c-528c-4296-8e2c-8dcccd333eb7" containerName="extract"
Jul 27 18:35:08.220648 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:08.220584    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="065a521c-528c-4296-8e2c-8dcccd333eb7" containerName="pull"
Jul 27 18:35:08.220648 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:08.220590    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="065a521c-528c-4296-8e2c-8dcccd333eb7" containerName="pull"
Jul 27 18:35:08.220648 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:08.220603    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="065a521c-528c-4296-8e2c-8dcccd333eb7" containerName="util"
Jul 27 18:35:08.220648 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:08.220607    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="065a521c-528c-4296-8e2c-8dcccd333eb7" containerName="util"
Jul 27 18:35:08.220826 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:08.220660    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="065a521c-528c-4296-8e2c-8dcccd333eb7" containerName="extract"
Jul 27 18:35:08.222858 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:08.222834    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-lcncj"
Jul 27 18:35:08.225566 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:08.225548    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-kstmc\""
Jul 27 18:35:08.234057 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:08.234033    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lcncj"]
Jul 27 18:35:08.313171 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:08.313134    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5ht8\" (UniqueName: \"kubernetes.io/projected/4c2a86e5-15ae-4abe-8886-339e8bd45197-kube-api-access-c5ht8\") pod \"cert-manager-55b4d954d5-lcncj\" (UID: \"4c2a86e5-15ae-4abe-8886-339e8bd45197\") " pod="cert-manager/cert-manager-55b4d954d5-lcncj"
Jul 27 18:35:08.313343 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:08.313236    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c2a86e5-15ae-4abe-8886-339e8bd45197-bound-sa-token\") pod \"cert-manager-55b4d954d5-lcncj\" (UID: \"4c2a86e5-15ae-4abe-8886-339e8bd45197\") " pod="cert-manager/cert-manager-55b4d954d5-lcncj"
Jul 27 18:35:08.413855 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:08.413817    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c2a86e5-15ae-4abe-8886-339e8bd45197-bound-sa-token\") pod \"cert-manager-55b4d954d5-lcncj\" (UID: \"4c2a86e5-15ae-4abe-8886-339e8bd45197\") " pod="cert-manager/cert-manager-55b4d954d5-lcncj"
Jul 27 18:35:08.414022 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:08.413883    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5ht8\" (UniqueName: \"kubernetes.io/projected/4c2a86e5-15ae-4abe-8886-339e8bd45197-kube-api-access-c5ht8\") pod \"cert-manager-55b4d954d5-lcncj\" (UID: \"4c2a86e5-15ae-4abe-8886-339e8bd45197\") " pod="cert-manager/cert-manager-55b4d954d5-lcncj"
Jul 27 18:35:08.422814 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:08.422784    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c2a86e5-15ae-4abe-8886-339e8bd45197-bound-sa-token\") pod \"cert-manager-55b4d954d5-lcncj\" (UID: \"4c2a86e5-15ae-4abe-8886-339e8bd45197\") " pod="cert-manager/cert-manager-55b4d954d5-lcncj"
Jul 27 18:35:08.422923 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:08.422791    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5ht8\" (UniqueName: \"kubernetes.io/projected/4c2a86e5-15ae-4abe-8886-339e8bd45197-kube-api-access-c5ht8\") pod \"cert-manager-55b4d954d5-lcncj\" (UID: \"4c2a86e5-15ae-4abe-8886-339e8bd45197\") " pod="cert-manager/cert-manager-55b4d954d5-lcncj"
Jul 27 18:35:08.531637 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:08.531608    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-lcncj"
Jul 27 18:35:08.666636 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:08.666615    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lcncj"]
Jul 27 18:35:08.668896 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:35:08.668863    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c2a86e5_15ae_4abe_8886_339e8bd45197.slice/crio-3c25e4f873715feaf47ef561bc44dbefd6d276545ec282b590d370c402a5cf7b WatchSource:0}: Error finding container 3c25e4f873715feaf47ef561bc44dbefd6d276545ec282b590d370c402a5cf7b: Status 404 returned error can't find the container with id 3c25e4f873715feaf47ef561bc44dbefd6d276545ec282b590d370c402a5cf7b
Jul 27 18:35:08.787542 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:08.787465    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-lcncj" event={"ID":"4c2a86e5-15ae-4abe-8886-339e8bd45197","Type":"ContainerStarted","Data":"13c3d30adea794588f758d445df5e663f96eae5a22f47fc55f642f68d7e5c3bf"}
Jul 27 18:35:08.787542 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:08.787504    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-lcncj" event={"ID":"4c2a86e5-15ae-4abe-8886-339e8bd45197","Type":"ContainerStarted","Data":"3c25e4f873715feaf47ef561bc44dbefd6d276545ec282b590d370c402a5cf7b"}
Jul 27 18:35:08.805390 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:08.805303    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-lcncj" podStartSLOduration=0.805282797 podStartE2EDuration="805.282797ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:35:08.804805464 +0000 UTC m=+369.239063076" watchObservedRunningTime="2026-07-27 18:35:08.805282797 +0000 UTC m=+369.239540409"
Jul 27 18:35:21.076213 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:21.076181    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc"]
Jul 27 18:35:21.078819 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:21.078803    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc"
Jul 27 18:35:21.082312 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:21.082281    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 27 18:35:21.082417 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:21.082391    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 27 18:35:21.083608 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:21.083595    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-s2xv2\""
Jul 27 18:35:21.093769 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:21.093743    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc"]
Jul 27 18:35:21.120344 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:21.120309    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05c8838c-5296-4646-baf5-13c18a2566c0-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc\" (UID: \"05c8838c-5296-4646-baf5-13c18a2566c0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc"
Jul 27 18:35:21.120487 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:21.120353    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05c8838c-5296-4646-baf5-13c18a2566c0-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc\" (UID: \"05c8838c-5296-4646-baf5-13c18a2566c0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc"
Jul 27 18:35:21.120487 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:21.120381    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d48f6\" (UniqueName: \"kubernetes.io/projected/05c8838c-5296-4646-baf5-13c18a2566c0-kube-api-access-d48f6\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc\" (UID: \"05c8838c-5296-4646-baf5-13c18a2566c0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc"
Jul 27 18:35:21.220750 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:21.220709    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d48f6\" (UniqueName: \"kubernetes.io/projected/05c8838c-5296-4646-baf5-13c18a2566c0-kube-api-access-d48f6\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc\" (UID: \"05c8838c-5296-4646-baf5-13c18a2566c0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc"
Jul 27 18:35:21.220933 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:21.220795    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05c8838c-5296-4646-baf5-13c18a2566c0-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc\" (UID: \"05c8838c-5296-4646-baf5-13c18a2566c0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc"
Jul 27 18:35:21.220933 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:21.220843    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05c8838c-5296-4646-baf5-13c18a2566c0-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc\" (UID: \"05c8838c-5296-4646-baf5-13c18a2566c0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc"
Jul 27 18:35:21.221300 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:21.221280    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05c8838c-5296-4646-baf5-13c18a2566c0-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc\" (UID: \"05c8838c-5296-4646-baf5-13c18a2566c0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc"
Jul 27 18:35:21.221372 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:21.221304    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05c8838c-5296-4646-baf5-13c18a2566c0-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc\" (UID: \"05c8838c-5296-4646-baf5-13c18a2566c0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc"
Jul 27 18:35:21.229798 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:21.229775    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d48f6\" (UniqueName: \"kubernetes.io/projected/05c8838c-5296-4646-baf5-13c18a2566c0-kube-api-access-d48f6\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc\" (UID: \"05c8838c-5296-4646-baf5-13c18a2566c0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc"
Jul 27 18:35:21.388471 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:21.388389    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc"
Jul 27 18:35:21.519722 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:21.519698    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc"]
Jul 27 18:35:21.521902 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:35:21.521873    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c8838c_5296_4646_baf5_13c18a2566c0.slice/crio-6dcc1bbcb8c66e66efe09667c386785a998601bed126a6af38e3e3128966a868 WatchSource:0}: Error finding container 6dcc1bbcb8c66e66efe09667c386785a998601bed126a6af38e3e3128966a868: Status 404 returned error can't find the container with id 6dcc1bbcb8c66e66efe09667c386785a998601bed126a6af38e3e3128966a868
Jul 27 18:35:21.847698 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:21.847665    2566 generic.go:356] "Generic (PLEG): container finished" podID="05c8838c-5296-4646-baf5-13c18a2566c0" containerID="92b024c0f6cde95d3f6db50a9844024b0de3bad30fb7019548023b27c10f2c20" exitCode=0
Jul 27 18:35:21.847909 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:21.847754    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc" event={"ID":"05c8838c-5296-4646-baf5-13c18a2566c0","Type":"ContainerDied","Data":"92b024c0f6cde95d3f6db50a9844024b0de3bad30fb7019548023b27c10f2c20"}
Jul 27 18:35:21.847909 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:21.847810    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc" event={"ID":"05c8838c-5296-4646-baf5-13c18a2566c0","Type":"ContainerStarted","Data":"6dcc1bbcb8c66e66efe09667c386785a998601bed126a6af38e3e3128966a868"}
Jul 27 18:35:22.854850 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:22.854774    2566 generic.go:356] "Generic (PLEG): container finished" podID="05c8838c-5296-4646-baf5-13c18a2566c0" containerID="d2461d60b94679797fd339da4002c4fd3905b601c6f607f8f5a42c66159ea7bc" exitCode=0
Jul 27 18:35:22.855224 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:22.854840    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc" event={"ID":"05c8838c-5296-4646-baf5-13c18a2566c0","Type":"ContainerDied","Data":"d2461d60b94679797fd339da4002c4fd3905b601c6f607f8f5a42c66159ea7bc"}
Jul 27 18:35:23.861711 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:23.861676    2566 generic.go:356] "Generic (PLEG): container finished" podID="05c8838c-5296-4646-baf5-13c18a2566c0" containerID="d96180a4b9392bc31b06d412cfa964ebb36301411ef7a7f9b77bc2f91ad79c5a" exitCode=0
Jul 27 18:35:23.862084 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:23.861735    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc" event={"ID":"05c8838c-5296-4646-baf5-13c18a2566c0","Type":"ContainerDied","Data":"d96180a4b9392bc31b06d412cfa964ebb36301411ef7a7f9b77bc2f91ad79c5a"}
Jul 27 18:35:24.993172 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:24.993143    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc"
Jul 27 18:35:25.052431 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:25.052405    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05c8838c-5296-4646-baf5-13c18a2566c0-bundle\") pod \"05c8838c-5296-4646-baf5-13c18a2566c0\" (UID: \"05c8838c-5296-4646-baf5-13c18a2566c0\") "
Jul 27 18:35:25.052590 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:25.052455    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05c8838c-5296-4646-baf5-13c18a2566c0-util\") pod \"05c8838c-5296-4646-baf5-13c18a2566c0\" (UID: \"05c8838c-5296-4646-baf5-13c18a2566c0\") "
Jul 27 18:35:25.052590 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:25.052497    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d48f6\" (UniqueName: \"kubernetes.io/projected/05c8838c-5296-4646-baf5-13c18a2566c0-kube-api-access-d48f6\") pod \"05c8838c-5296-4646-baf5-13c18a2566c0\" (UID: \"05c8838c-5296-4646-baf5-13c18a2566c0\") "
Jul 27 18:35:25.053329 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:25.053294    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c8838c-5296-4646-baf5-13c18a2566c0-bundle" (OuterVolumeSpecName: "bundle") pod "05c8838c-5296-4646-baf5-13c18a2566c0" (UID: "05c8838c-5296-4646-baf5-13c18a2566c0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:35:25.054782 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:25.054762    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c8838c-5296-4646-baf5-13c18a2566c0-kube-api-access-d48f6" (OuterVolumeSpecName: "kube-api-access-d48f6") pod "05c8838c-5296-4646-baf5-13c18a2566c0" (UID: "05c8838c-5296-4646-baf5-13c18a2566c0"). InnerVolumeSpecName "kube-api-access-d48f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:35:25.058577 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:25.058556    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c8838c-5296-4646-baf5-13c18a2566c0-util" (OuterVolumeSpecName: "util") pod "05c8838c-5296-4646-baf5-13c18a2566c0" (UID: "05c8838c-5296-4646-baf5-13c18a2566c0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:35:25.153245 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:25.153159    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05c8838c-5296-4646-baf5-13c18a2566c0-bundle\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:35:25.153245 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:25.153187    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05c8838c-5296-4646-baf5-13c18a2566c0-util\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:35:25.153245 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:25.153197    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d48f6\" (UniqueName: \"kubernetes.io/projected/05c8838c-5296-4646-baf5-13c18a2566c0-kube-api-access-d48f6\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:35:25.872681 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:25.872645    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc" event={"ID":"05c8838c-5296-4646-baf5-13c18a2566c0","Type":"ContainerDied","Data":"6dcc1bbcb8c66e66efe09667c386785a998601bed126a6af38e3e3128966a868"}
Jul 27 18:35:25.872847 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:25.872693    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdgsc"
Jul 27 18:35:25.872847 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:25.872700    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dcc1bbcb8c66e66efe09667c386785a998601bed126a6af38e3e3128966a868"
Jul 27 18:35:32.563981 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.563931    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57d99c84d4-6x7ln"]
Jul 27 18:35:32.567251 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.567213    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05c8838c-5296-4646-baf5-13c18a2566c0" containerName="util"
Jul 27 18:35:32.567251 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.567239    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="05c8838c-5296-4646-baf5-13c18a2566c0" containerName="util"
Jul 27 18:35:32.567438 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.567263    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05c8838c-5296-4646-baf5-13c18a2566c0" containerName="extract"
Jul 27 18:35:32.567438 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.567269    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="05c8838c-5296-4646-baf5-13c18a2566c0" containerName="extract"
Jul 27 18:35:32.567438 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.567276    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05c8838c-5296-4646-baf5-13c18a2566c0" containerName="pull"
Jul 27 18:35:32.567438 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.567281    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="05c8838c-5296-4646-baf5-13c18a2566c0" containerName="pull"
Jul 27 18:35:32.567438 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.567330    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="05c8838c-5296-4646-baf5-13c18a2566c0" containerName="extract"
Jul 27 18:35:32.570452 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.570429    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-6x7ln"
Jul 27 18:35:32.573948 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.573928    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 27 18:35:32.574393 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.574370    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 27 18:35:32.574537 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.574513    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 27 18:35:32.574586 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.574536    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dvvg4\""
Jul 27 18:35:32.576722 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.576706    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 27 18:35:32.591000 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.590968    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57d99c84d4-6x7ln"]
Jul 27 18:35:32.619838 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.619805    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f980df40-310c-4e8b-a9d7-f52ab379e938-tmp\") pod \"opendatahub-operator-controller-manager-57d99c84d4-6x7ln\" (UID: \"f980df40-310c-4e8b-a9d7-f52ab379e938\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-6x7ln"
Jul 27 18:35:32.619838 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.619836    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f980df40-310c-4e8b-a9d7-f52ab379e938-webhook-cert\") pod \"opendatahub-operator-controller-manager-57d99c84d4-6x7ln\" (UID: \"f980df40-310c-4e8b-a9d7-f52ab379e938\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-6x7ln"
Jul 27 18:35:32.620019 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.619868    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f980df40-310c-4e8b-a9d7-f52ab379e938-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57d99c84d4-6x7ln\" (UID: \"f980df40-310c-4e8b-a9d7-f52ab379e938\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-6x7ln"
Jul 27 18:35:32.620019 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.619923    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f980df40-310c-4e8b-a9d7-f52ab379e938-manifests\") pod \"opendatahub-operator-controller-manager-57d99c84d4-6x7ln\" (UID: \"f980df40-310c-4e8b-a9d7-f52ab379e938\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-6x7ln"
Jul 27 18:35:32.620019 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.619985    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x22j\" (UniqueName: \"kubernetes.io/projected/f980df40-310c-4e8b-a9d7-f52ab379e938-kube-api-access-4x22j\") pod \"opendatahub-operator-controller-manager-57d99c84d4-6x7ln\" (UID: \"f980df40-310c-4e8b-a9d7-f52ab379e938\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-6x7ln"
Jul 27 18:35:32.720434 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.720394    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x22j\" (UniqueName: \"kubernetes.io/projected/f980df40-310c-4e8b-a9d7-f52ab379e938-kube-api-access-4x22j\") pod \"opendatahub-operator-controller-manager-57d99c84d4-6x7ln\" (UID: \"f980df40-310c-4e8b-a9d7-f52ab379e938\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-6x7ln"
Jul 27 18:35:32.720631 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.720464    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f980df40-310c-4e8b-a9d7-f52ab379e938-tmp\") pod \"opendatahub-operator-controller-manager-57d99c84d4-6x7ln\" (UID: \"f980df40-310c-4e8b-a9d7-f52ab379e938\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-6x7ln"
Jul 27 18:35:32.720631 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.720483    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f980df40-310c-4e8b-a9d7-f52ab379e938-webhook-cert\") pod \"opendatahub-operator-controller-manager-57d99c84d4-6x7ln\" (UID: \"f980df40-310c-4e8b-a9d7-f52ab379e938\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-6x7ln"
Jul 27 18:35:32.720631 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.720508    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f980df40-310c-4e8b-a9d7-f52ab379e938-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57d99c84d4-6x7ln\" (UID: \"f980df40-310c-4e8b-a9d7-f52ab379e938\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-6x7ln"
Jul 27 18:35:32.720631 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.720536    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f980df40-310c-4e8b-a9d7-f52ab379e938-manifests\") pod \"opendatahub-operator-controller-manager-57d99c84d4-6x7ln\" (UID: \"f980df40-310c-4e8b-a9d7-f52ab379e938\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-6x7ln"
Jul 27 18:35:32.720946 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.720914    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f980df40-310c-4e8b-a9d7-f52ab379e938-manifests\") pod \"opendatahub-operator-controller-manager-57d99c84d4-6x7ln\" (UID: \"f980df40-310c-4e8b-a9d7-f52ab379e938\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-6x7ln"
Jul 27 18:35:32.722900 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.722872    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f980df40-310c-4e8b-a9d7-f52ab379e938-tmp\") pod \"opendatahub-operator-controller-manager-57d99c84d4-6x7ln\" (UID: \"f980df40-310c-4e8b-a9d7-f52ab379e938\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-6x7ln"
Jul 27 18:35:32.723027 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.723009    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f980df40-310c-4e8b-a9d7-f52ab379e938-webhook-cert\") pod \"opendatahub-operator-controller-manager-57d99c84d4-6x7ln\" (UID: \"f980df40-310c-4e8b-a9d7-f52ab379e938\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-6x7ln"
Jul 27 18:35:32.723090 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.723068    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f980df40-310c-4e8b-a9d7-f52ab379e938-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57d99c84d4-6x7ln\" (UID: \"f980df40-310c-4e8b-a9d7-f52ab379e938\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-6x7ln"
Jul 27 18:35:32.735510 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.735478    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x22j\" (UniqueName: \"kubernetes.io/projected/f980df40-310c-4e8b-a9d7-f52ab379e938-kube-api-access-4x22j\") pod \"opendatahub-operator-controller-manager-57d99c84d4-6x7ln\" (UID: \"f980df40-310c-4e8b-a9d7-f52ab379e938\") " pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-6x7ln"
Jul 27 18:35:32.881778 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:32.881692    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-6x7ln"
Jul 27 18:35:33.020512 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:33.020481    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57d99c84d4-6x7ln"]
Jul 27 18:35:33.023348 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:35:33.023321    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf980df40_310c_4e8b_a9d7_f52ab379e938.slice/crio-0cefe86c712efbdd95ce64cfd0813ce62d53f5ed757e3f325379e47a2ff488f7 WatchSource:0}: Error finding container 0cefe86c712efbdd95ce64cfd0813ce62d53f5ed757e3f325379e47a2ff488f7: Status 404 returned error can't find the container with id 0cefe86c712efbdd95ce64cfd0813ce62d53f5ed757e3f325379e47a2ff488f7
Jul 27 18:35:33.911197 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:33.911163    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-6x7ln" event={"ID":"f980df40-310c-4e8b-a9d7-f52ab379e938","Type":"ContainerStarted","Data":"0cefe86c712efbdd95ce64cfd0813ce62d53f5ed757e3f325379e47a2ff488f7"}
Jul 27 18:35:37.934319 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:37.934287    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-6x7ln" event={"ID":"f980df40-310c-4e8b-a9d7-f52ab379e938","Type":"ContainerStarted","Data":"b287f2c4f44f69e89ce8e3fad9effc2e28b5317ad9757117108833b520432c3a"}
Jul 27 18:35:38.940318 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:38.940284    2566 generic.go:356] "Generic (PLEG): container finished" podID="f980df40-310c-4e8b-a9d7-f52ab379e938" containerID="b287f2c4f44f69e89ce8e3fad9effc2e28b5317ad9757117108833b520432c3a" exitCode=0
Jul 27 18:35:38.940756 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:38.940331    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-6x7ln" event={"ID":"f980df40-310c-4e8b-a9d7-f52ab379e938","Type":"ContainerDied","Data":"b287f2c4f44f69e89ce8e3fad9effc2e28b5317ad9757117108833b520432c3a"}
Jul 27 18:35:39.947169 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:39.947068    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-6x7ln" event={"ID":"f980df40-310c-4e8b-a9d7-f52ab379e938","Type":"ContainerStarted","Data":"cb0a23b1e9c2e83aec696752195adeab1db8358062289a31fc743f2da23de5cc"}
Jul 27 18:35:39.947510 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:39.947225    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-6x7ln"
Jul 27 18:35:39.982929 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:39.982873    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-6x7ln" podStartSLOduration=2.338905795 podStartE2EDuration="7.982852194s" totalImagesPullingTime="4.601606328s" totalInitContainerRuntime="1.042340071s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:35:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:35:39.981383135 +0000 UTC m=+400.415640747" watchObservedRunningTime="2026-07-27 18:35:39.982852194 +0000 UTC m=+400.417109806"
Jul 27 18:35:50.954088 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:50.954055    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-57d99c84d4-6x7ln"
Jul 27 18:35:53.119579 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.119543    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cc45b746b-rwfq7"]
Jul 27 18:35:53.122719 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.122703    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cc45b746b-rwfq7"
Jul 27 18:35:53.127307 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.127285    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 27 18:35:53.127419 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.127373    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xr9vw\""
Jul 27 18:35:53.127419 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.127286    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 27 18:35:53.127419 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.127399    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 27 18:35:53.127677 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.127660    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 27 18:35:53.127677 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.127668    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 27 18:35:53.135751 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.135732    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cc45b746b-rwfq7"]
Jul 27 18:35:53.298260 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.298207    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0c986fb5-995a-4218-9879-19baef748503-manager-config\") pod \"lws-controller-manager-7cc45b746b-rwfq7\" (UID: \"0c986fb5-995a-4218-9879-19baef748503\") " pod="openshift-lws-operator/lws-controller-manager-7cc45b746b-rwfq7"
Jul 27 18:35:53.298442 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.298362    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwgvh\" (UniqueName: \"kubernetes.io/projected/0c986fb5-995a-4218-9879-19baef748503-kube-api-access-lwgvh\") pod \"lws-controller-manager-7cc45b746b-rwfq7\" (UID: \"0c986fb5-995a-4218-9879-19baef748503\") " pod="openshift-lws-operator/lws-controller-manager-7cc45b746b-rwfq7"
Jul 27 18:35:53.298442 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.298401    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c986fb5-995a-4218-9879-19baef748503-metrics-cert\") pod \"lws-controller-manager-7cc45b746b-rwfq7\" (UID: \"0c986fb5-995a-4218-9879-19baef748503\") " pod="openshift-lws-operator/lws-controller-manager-7cc45b746b-rwfq7"
Jul 27 18:35:53.298525 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.298446    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c986fb5-995a-4218-9879-19baef748503-cert\") pod \"lws-controller-manager-7cc45b746b-rwfq7\" (UID: \"0c986fb5-995a-4218-9879-19baef748503\") " pod="openshift-lws-operator/lws-controller-manager-7cc45b746b-rwfq7"
Jul 27 18:35:53.399432 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.399345    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0c986fb5-995a-4218-9879-19baef748503-manager-config\") pod \"lws-controller-manager-7cc45b746b-rwfq7\" (UID: \"0c986fb5-995a-4218-9879-19baef748503\") " pod="openshift-lws-operator/lws-controller-manager-7cc45b746b-rwfq7"
Jul 27 18:35:53.399432 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.399411    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwgvh\" (UniqueName: \"kubernetes.io/projected/0c986fb5-995a-4218-9879-19baef748503-kube-api-access-lwgvh\") pod \"lws-controller-manager-7cc45b746b-rwfq7\" (UID: \"0c986fb5-995a-4218-9879-19baef748503\") " pod="openshift-lws-operator/lws-controller-manager-7cc45b746b-rwfq7"
Jul 27 18:35:53.399432 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.399433    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c986fb5-995a-4218-9879-19baef748503-metrics-cert\") pod \"lws-controller-manager-7cc45b746b-rwfq7\" (UID: \"0c986fb5-995a-4218-9879-19baef748503\") " pod="openshift-lws-operator/lws-controller-manager-7cc45b746b-rwfq7"
Jul 27 18:35:53.399670 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.399475    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c986fb5-995a-4218-9879-19baef748503-cert\") pod \"lws-controller-manager-7cc45b746b-rwfq7\" (UID: \"0c986fb5-995a-4218-9879-19baef748503\") " pod="openshift-lws-operator/lws-controller-manager-7cc45b746b-rwfq7"
Jul 27 18:35:53.400045 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.400008    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0c986fb5-995a-4218-9879-19baef748503-manager-config\") pod \"lws-controller-manager-7cc45b746b-rwfq7\" (UID: \"0c986fb5-995a-4218-9879-19baef748503\") " pod="openshift-lws-operator/lws-controller-manager-7cc45b746b-rwfq7"
Jul 27 18:35:53.402079 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.402061    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c986fb5-995a-4218-9879-19baef748503-cert\") pod \"lws-controller-manager-7cc45b746b-rwfq7\" (UID: \"0c986fb5-995a-4218-9879-19baef748503\") " pod="openshift-lws-operator/lws-controller-manager-7cc45b746b-rwfq7"
Jul 27 18:35:53.402218 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.402201    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c986fb5-995a-4218-9879-19baef748503-metrics-cert\") pod \"lws-controller-manager-7cc45b746b-rwfq7\" (UID: \"0c986fb5-995a-4218-9879-19baef748503\") " pod="openshift-lws-operator/lws-controller-manager-7cc45b746b-rwfq7"
Jul 27 18:35:53.408527 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.408503    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwgvh\" (UniqueName: \"kubernetes.io/projected/0c986fb5-995a-4218-9879-19baef748503-kube-api-access-lwgvh\") pod \"lws-controller-manager-7cc45b746b-rwfq7\" (UID: \"0c986fb5-995a-4218-9879-19baef748503\") " pod="openshift-lws-operator/lws-controller-manager-7cc45b746b-rwfq7"
Jul 27 18:35:53.431499 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.431475    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cc45b746b-rwfq7"
Jul 27 18:35:53.536879 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.536848    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7896769b64-dmt58"]
Jul 27 18:35:53.540379 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.540358    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7896769b64-dmt58"
Jul 27 18:35:53.543192 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.543171    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 27 18:35:53.543789 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.543770    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 27 18:35:53.543965 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.543941    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-rg85g\""
Jul 27 18:35:53.544049 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.543966    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 27 18:35:53.544049 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.543987    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 27 18:35:53.554571 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.554549    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7896769b64-dmt58"]
Jul 27 18:35:53.570915 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.570890    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cc45b746b-rwfq7"]
Jul 27 18:35:53.571882 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:35:53.571860    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c986fb5_995a_4218_9879_19baef748503.slice/crio-f1311f39c3a9eef561c5635869bc10f0b6771caf3a39aa10afd4907cf37ca36b WatchSource:0}: Error finding container f1311f39c3a9eef561c5635869bc10f0b6771caf3a39aa10afd4907cf37ca36b: Status 404 returned error can't find the container with id f1311f39c3a9eef561c5635869bc10f0b6771caf3a39aa10afd4907cf37ca36b
Jul 27 18:35:53.703025 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.702939    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnqfx\" (UniqueName: \"kubernetes.io/projected/445cd3bc-555e-47cf-99c7-ca02cdd89d85-kube-api-access-wnqfx\") pod \"kube-auth-proxy-7896769b64-dmt58\" (UID: \"445cd3bc-555e-47cf-99c7-ca02cdd89d85\") " pod="openshift-ingress/kube-auth-proxy-7896769b64-dmt58"
Jul 27 18:35:53.703025 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.702981    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/445cd3bc-555e-47cf-99c7-ca02cdd89d85-tls-certs\") pod \"kube-auth-proxy-7896769b64-dmt58\" (UID: \"445cd3bc-555e-47cf-99c7-ca02cdd89d85\") " pod="openshift-ingress/kube-auth-proxy-7896769b64-dmt58"
Jul 27 18:35:53.703025 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.703016    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/445cd3bc-555e-47cf-99c7-ca02cdd89d85-tmp\") pod \"kube-auth-proxy-7896769b64-dmt58\" (UID: \"445cd3bc-555e-47cf-99c7-ca02cdd89d85\") " pod="openshift-ingress/kube-auth-proxy-7896769b64-dmt58"
Jul 27 18:35:53.804422 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.804385    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/445cd3bc-555e-47cf-99c7-ca02cdd89d85-tls-certs\") pod \"kube-auth-proxy-7896769b64-dmt58\" (UID: \"445cd3bc-555e-47cf-99c7-ca02cdd89d85\") " pod="openshift-ingress/kube-auth-proxy-7896769b64-dmt58"
Jul 27 18:35:53.804595 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.804466    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/445cd3bc-555e-47cf-99c7-ca02cdd89d85-tmp\") pod \"kube-auth-proxy-7896769b64-dmt58\" (UID: \"445cd3bc-555e-47cf-99c7-ca02cdd89d85\") " pod="openshift-ingress/kube-auth-proxy-7896769b64-dmt58"
Jul 27 18:35:53.804595 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:35:53.804532    2566 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 27 18:35:53.804595 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.804544    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnqfx\" (UniqueName: \"kubernetes.io/projected/445cd3bc-555e-47cf-99c7-ca02cdd89d85-kube-api-access-wnqfx\") pod \"kube-auth-proxy-7896769b64-dmt58\" (UID: \"445cd3bc-555e-47cf-99c7-ca02cdd89d85\") " pod="openshift-ingress/kube-auth-proxy-7896769b64-dmt58"
Jul 27 18:35:53.804595 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:35:53.804589    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/445cd3bc-555e-47cf-99c7-ca02cdd89d85-tls-certs podName:445cd3bc-555e-47cf-99c7-ca02cdd89d85 nodeName:}" failed. No retries permitted until 2026-07-27 18:35:54.304573696 +0000 UTC m=+414.738831287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/445cd3bc-555e-47cf-99c7-ca02cdd89d85-tls-certs") pod "kube-auth-proxy-7896769b64-dmt58" (UID: "445cd3bc-555e-47cf-99c7-ca02cdd89d85") : secret "kube-auth-proxy-tls" not found
Jul 27 18:35:53.806886 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.806869    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/445cd3bc-555e-47cf-99c7-ca02cdd89d85-tmp\") pod \"kube-auth-proxy-7896769b64-dmt58\" (UID: \"445cd3bc-555e-47cf-99c7-ca02cdd89d85\") " pod="openshift-ingress/kube-auth-proxy-7896769b64-dmt58"
Jul 27 18:35:53.829253 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:53.829225    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnqfx\" (UniqueName: \"kubernetes.io/projected/445cd3bc-555e-47cf-99c7-ca02cdd89d85-kube-api-access-wnqfx\") pod \"kube-auth-proxy-7896769b64-dmt58\" (UID: \"445cd3bc-555e-47cf-99c7-ca02cdd89d85\") " pod="openshift-ingress/kube-auth-proxy-7896769b64-dmt58"
Jul 27 18:35:54.015664 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:54.015582    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cc45b746b-rwfq7" event={"ID":"0c986fb5-995a-4218-9879-19baef748503","Type":"ContainerStarted","Data":"f1311f39c3a9eef561c5635869bc10f0b6771caf3a39aa10afd4907cf37ca36b"}
Jul 27 18:35:54.309955 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:54.309925    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/445cd3bc-555e-47cf-99c7-ca02cdd89d85-tls-certs\") pod \"kube-auth-proxy-7896769b64-dmt58\" (UID: \"445cd3bc-555e-47cf-99c7-ca02cdd89d85\") " pod="openshift-ingress/kube-auth-proxy-7896769b64-dmt58"
Jul 27 18:35:54.312282 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:54.312255    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/445cd3bc-555e-47cf-99c7-ca02cdd89d85-tls-certs\") pod \"kube-auth-proxy-7896769b64-dmt58\" (UID: \"445cd3bc-555e-47cf-99c7-ca02cdd89d85\") " pod="openshift-ingress/kube-auth-proxy-7896769b64-dmt58"
Jul 27 18:35:54.450541 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:54.450508    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7896769b64-dmt58"
Jul 27 18:35:54.590677 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:54.590599    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7896769b64-dmt58"]
Jul 27 18:35:54.594414 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:35:54.594385    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod445cd3bc_555e_47cf_99c7_ca02cdd89d85.slice/crio-80510cd42dc3614abc10229eefe56d3c5f19d847c77420993310871260276da8 WatchSource:0}: Error finding container 80510cd42dc3614abc10229eefe56d3c5f19d847c77420993310871260276da8: Status 404 returned error can't find the container with id 80510cd42dc3614abc10229eefe56d3c5f19d847c77420993310871260276da8
Jul 27 18:35:55.022226 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:55.022104    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7896769b64-dmt58" event={"ID":"445cd3bc-555e-47cf-99c7-ca02cdd89d85","Type":"ContainerStarted","Data":"80510cd42dc3614abc10229eefe56d3c5f19d847c77420993310871260276da8"}
Jul 27 18:35:58.045767 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:58.045729    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7896769b64-dmt58" event={"ID":"445cd3bc-555e-47cf-99c7-ca02cdd89d85","Type":"ContainerStarted","Data":"719c531b47dfcd42cfdb9d3d23eabe71b7252198f260a65cfa8438b369cf7720"}
Jul 27 18:35:58.068760 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:35:58.068711    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7896769b64-dmt58" podStartSLOduration=2.336140793 podStartE2EDuration="5.068696282s" totalImagesPullingTime="2.732555489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:35:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:35:58.067285297 +0000 UTC m=+418.501542908" watchObservedRunningTime="2026-07-27 18:35:58.068696282 +0000 UTC m=+418.502953892"
Jul 27 18:36:00.057790 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:36:00.057707    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cc45b746b-rwfq7" event={"ID":"0c986fb5-995a-4218-9879-19baef748503","Type":"ContainerStarted","Data":"eb5a70fab3b4d6ab15ce5d8ec9f1962c9385f8fff471717270a05830c3fead96"}
Jul 27 18:36:00.058114 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:36:00.057791    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7cc45b746b-rwfq7"
Jul 27 18:36:00.076486 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:36:00.076434    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7cc45b746b-rwfq7" podStartSLOduration=0.853712435 podStartE2EDuration="7.076416873s" totalImagesPullingTime="6.222704438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:35:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:36:00.075667117 +0000 UTC m=+420.509924789" watchObservedRunningTime="2026-07-27 18:36:00.076416873 +0000 UTC m=+420.510674484"
Jul 27 18:36:11.065025 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:36:11.064989    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7cc45b746b-rwfq7"
Jul 27 18:37:02.296609 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:02.296577    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cb99d6978-pctpl"]
Jul 27 18:37:02.299033 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:02.299015    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cb99d6978-pctpl"
Jul 27 18:37:02.313291 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:02.313270    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cb99d6978-pctpl"]
Jul 27 18:37:02.379088 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:02.379060    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1103b1ea-cab3-4559-aa88-fd5743f33a8d-oauth-serving-cert\") pod \"console-7cb99d6978-pctpl\" (UID: \"1103b1ea-cab3-4559-aa88-fd5743f33a8d\") " pod="openshift-console/console-7cb99d6978-pctpl"
Jul 27 18:37:02.379259 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:02.379094    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1103b1ea-cab3-4559-aa88-fd5743f33a8d-console-config\") pod \"console-7cb99d6978-pctpl\" (UID: \"1103b1ea-cab3-4559-aa88-fd5743f33a8d\") " pod="openshift-console/console-7cb99d6978-pctpl"
Jul 27 18:37:02.379259 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:02.379147    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1103b1ea-cab3-4559-aa88-fd5743f33a8d-console-oauth-config\") pod \"console-7cb99d6978-pctpl\" (UID: \"1103b1ea-cab3-4559-aa88-fd5743f33a8d\") " pod="openshift-console/console-7cb99d6978-pctpl"
Jul 27 18:37:02.379259 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:02.379189    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1103b1ea-cab3-4559-aa88-fd5743f33a8d-trusted-ca-bundle\") pod \"console-7cb99d6978-pctpl\" (UID: \"1103b1ea-cab3-4559-aa88-fd5743f33a8d\") " pod="openshift-console/console-7cb99d6978-pctpl"
Jul 27 18:37:02.379259 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:02.379242    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1103b1ea-cab3-4559-aa88-fd5743f33a8d-console-serving-cert\") pod \"console-7cb99d6978-pctpl\" (UID: \"1103b1ea-cab3-4559-aa88-fd5743f33a8d\") " pod="openshift-console/console-7cb99d6978-pctpl"
Jul 27 18:37:02.379420 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:02.379279    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbjgg\" (UniqueName: \"kubernetes.io/projected/1103b1ea-cab3-4559-aa88-fd5743f33a8d-kube-api-access-bbjgg\") pod \"console-7cb99d6978-pctpl\" (UID: \"1103b1ea-cab3-4559-aa88-fd5743f33a8d\") " pod="openshift-console/console-7cb99d6978-pctpl"
Jul 27 18:37:02.379420 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:02.379316    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1103b1ea-cab3-4559-aa88-fd5743f33a8d-service-ca\") pod \"console-7cb99d6978-pctpl\" (UID: \"1103b1ea-cab3-4559-aa88-fd5743f33a8d\") " pod="openshift-console/console-7cb99d6978-pctpl"
Jul 27 18:37:02.479719 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:02.479686    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1103b1ea-cab3-4559-aa88-fd5743f33a8d-trusted-ca-bundle\") pod \"console-7cb99d6978-pctpl\" (UID: \"1103b1ea-cab3-4559-aa88-fd5743f33a8d\") " pod="openshift-console/console-7cb99d6978-pctpl"
Jul 27 18:37:02.479719 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:02.479723    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1103b1ea-cab3-4559-aa88-fd5743f33a8d-console-serving-cert\") pod \"console-7cb99d6978-pctpl\" (UID: \"1103b1ea-cab3-4559-aa88-fd5743f33a8d\") " pod="openshift-console/console-7cb99d6978-pctpl"
Jul 27 18:37:02.479941 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:02.479750    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbjgg\" (UniqueName: \"kubernetes.io/projected/1103b1ea-cab3-4559-aa88-fd5743f33a8d-kube-api-access-bbjgg\") pod \"console-7cb99d6978-pctpl\" (UID: \"1103b1ea-cab3-4559-aa88-fd5743f33a8d\") " pod="openshift-console/console-7cb99d6978-pctpl"
Jul 27 18:37:02.479941 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:02.479783    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1103b1ea-cab3-4559-aa88-fd5743f33a8d-service-ca\") pod \"console-7cb99d6978-pctpl\" (UID: \"1103b1ea-cab3-4559-aa88-fd5743f33a8d\") " pod="openshift-console/console-7cb99d6978-pctpl"
Jul 27 18:37:02.479941 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:02.479805    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1103b1ea-cab3-4559-aa88-fd5743f33a8d-oauth-serving-cert\") pod \"console-7cb99d6978-pctpl\" (UID: \"1103b1ea-cab3-4559-aa88-fd5743f33a8d\") " pod="openshift-console/console-7cb99d6978-pctpl"
Jul 27 18:37:02.479941 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:02.479819    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1103b1ea-cab3-4559-aa88-fd5743f33a8d-console-config\") pod \"console-7cb99d6978-pctpl\" (UID: \"1103b1ea-cab3-4559-aa88-fd5743f33a8d\") " pod="openshift-console/console-7cb99d6978-pctpl"
Jul 27 18:37:02.479941 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:02.479838    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1103b1ea-cab3-4559-aa88-fd5743f33a8d-console-oauth-config\") pod \"console-7cb99d6978-pctpl\" (UID: \"1103b1ea-cab3-4559-aa88-fd5743f33a8d\") " pod="openshift-console/console-7cb99d6978-pctpl"
Jul 27 18:37:02.480661 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:02.480623    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1103b1ea-cab3-4559-aa88-fd5743f33a8d-oauth-serving-cert\") pod \"console-7cb99d6978-pctpl\" (UID: \"1103b1ea-cab3-4559-aa88-fd5743f33a8d\") " pod="openshift-console/console-7cb99d6978-pctpl"
Jul 27 18:37:02.480778 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:02.480660    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1103b1ea-cab3-4559-aa88-fd5743f33a8d-console-config\") pod \"console-7cb99d6978-pctpl\" (UID: \"1103b1ea-cab3-4559-aa88-fd5743f33a8d\") " pod="openshift-console/console-7cb99d6978-pctpl"
Jul 27 18:37:02.480778 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:02.480687    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1103b1ea-cab3-4559-aa88-fd5743f33a8d-trusted-ca-bundle\") pod \"console-7cb99d6978-pctpl\" (UID: \"1103b1ea-cab3-4559-aa88-fd5743f33a8d\") " pod="openshift-console/console-7cb99d6978-pctpl"
Jul 27 18:37:02.480853 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:02.480812    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1103b1ea-cab3-4559-aa88-fd5743f33a8d-service-ca\") pod \"console-7cb99d6978-pctpl\" (UID: \"1103b1ea-cab3-4559-aa88-fd5743f33a8d\") " pod="openshift-console/console-7cb99d6978-pctpl"
Jul 27 18:37:02.482372 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:02.482345    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1103b1ea-cab3-4559-aa88-fd5743f33a8d-console-serving-cert\") pod \"console-7cb99d6978-pctpl\" (UID: \"1103b1ea-cab3-4559-aa88-fd5743f33a8d\") " pod="openshift-console/console-7cb99d6978-pctpl"
Jul 27 18:37:02.482486 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:02.482398    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1103b1ea-cab3-4559-aa88-fd5743f33a8d-console-oauth-config\") pod \"console-7cb99d6978-pctpl\" (UID: \"1103b1ea-cab3-4559-aa88-fd5743f33a8d\") " pod="openshift-console/console-7cb99d6978-pctpl"
Jul 27 18:37:02.488524 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:02.488503    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbjgg\" (UniqueName: \"kubernetes.io/projected/1103b1ea-cab3-4559-aa88-fd5743f33a8d-kube-api-access-bbjgg\") pod \"console-7cb99d6978-pctpl\" (UID: \"1103b1ea-cab3-4559-aa88-fd5743f33a8d\") " pod="openshift-console/console-7cb99d6978-pctpl"
Jul 27 18:37:02.609006 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:02.608915    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cb99d6978-pctpl"
Jul 27 18:37:02.746403 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:02.746379    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cb99d6978-pctpl"]
Jul 27 18:37:02.748477 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:37:02.748450    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1103b1ea_cab3_4559_aa88_fd5743f33a8d.slice/crio-141c1decd7f9996c7fc7dc8d25d35dae7956bba94af8675444ea2580d416d4a1 WatchSource:0}: Error finding container 141c1decd7f9996c7fc7dc8d25d35dae7956bba94af8675444ea2580d416d4a1: Status 404 returned error can't find the container with id 141c1decd7f9996c7fc7dc8d25d35dae7956bba94af8675444ea2580d416d4a1
Jul 27 18:37:03.356633 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:03.356602    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cb99d6978-pctpl" event={"ID":"1103b1ea-cab3-4559-aa88-fd5743f33a8d","Type":"ContainerStarted","Data":"ec7934d2d21dc5f0373c4e82a98a7f20db1001000efd4070eee4d9493de6c3db"}
Jul 27 18:37:03.357104 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:03.356648    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cb99d6978-pctpl" event={"ID":"1103b1ea-cab3-4559-aa88-fd5743f33a8d","Type":"ContainerStarted","Data":"141c1decd7f9996c7fc7dc8d25d35dae7956bba94af8675444ea2580d416d4a1"}
Jul 27 18:37:03.380301 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:03.380255    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7cb99d6978-pctpl" podStartSLOduration=1.38024084 podStartE2EDuration="1.38024084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:37:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:37:03.378318381 +0000 UTC m=+483.812575992" watchObservedRunningTime="2026-07-27 18:37:03.38024084 +0000 UTC m=+483.814498454"
Jul 27 18:37:10.162327 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.162297    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7"]
Jul 27 18:37:10.165075 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.165060    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7"
Jul 27 18:37:10.167910 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.167892    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 27 18:37:10.168041 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.167917    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jv8qx\""
Jul 27 18:37:10.168041 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.167924    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 27 18:37:10.174778 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.174758    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7"]
Jul 27 18:37:10.248466 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.248432    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0ccd514-6a2b-405f-a724-297b04ae55a7-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7\" (UID: \"a0ccd514-6a2b-405f-a724-297b04ae55a7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7"
Jul 27 18:37:10.248655 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.248507    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzwcb\" (UniqueName: \"kubernetes.io/projected/a0ccd514-6a2b-405f-a724-297b04ae55a7-kube-api-access-gzwcb\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7\" (UID: \"a0ccd514-6a2b-405f-a724-297b04ae55a7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7"
Jul 27 18:37:10.248655 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.248571    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0ccd514-6a2b-405f-a724-297b04ae55a7-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7\" (UID: \"a0ccd514-6a2b-405f-a724-297b04ae55a7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7"
Jul 27 18:37:10.349964 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.349928    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0ccd514-6a2b-405f-a724-297b04ae55a7-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7\" (UID: \"a0ccd514-6a2b-405f-a724-297b04ae55a7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7"
Jul 27 18:37:10.350170 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.349981    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzwcb\" (UniqueName: \"kubernetes.io/projected/a0ccd514-6a2b-405f-a724-297b04ae55a7-kube-api-access-gzwcb\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7\" (UID: \"a0ccd514-6a2b-405f-a724-297b04ae55a7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7"
Jul 27 18:37:10.350170 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.350021    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0ccd514-6a2b-405f-a724-297b04ae55a7-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7\" (UID: \"a0ccd514-6a2b-405f-a724-297b04ae55a7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7"
Jul 27 18:37:10.350375 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.350357    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0ccd514-6a2b-405f-a724-297b04ae55a7-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7\" (UID: \"a0ccd514-6a2b-405f-a724-297b04ae55a7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7"
Jul 27 18:37:10.350411 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.350380    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0ccd514-6a2b-405f-a724-297b04ae55a7-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7\" (UID: \"a0ccd514-6a2b-405f-a724-297b04ae55a7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7"
Jul 27 18:37:10.359184 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.359152    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzwcb\" (UniqueName: \"kubernetes.io/projected/a0ccd514-6a2b-405f-a724-297b04ae55a7-kube-api-access-gzwcb\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7\" (UID: \"a0ccd514-6a2b-405f-a724-297b04ae55a7\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7"
Jul 27 18:37:10.475205 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.475099    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7"
Jul 27 18:37:10.615033 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.615002    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7"]
Jul 27 18:37:10.616494 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:37:10.616467    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ccd514_6a2b_405f_a724_297b04ae55a7.slice/crio-6c25228f4e688145f71d2dac31b92c623a0a67cefe6447c343f9c953ab6a0cc0 WatchSource:0}: Error finding container 6c25228f4e688145f71d2dac31b92c623a0a67cefe6447c343f9c953ab6a0cc0: Status 404 returned error can't find the container with id 6c25228f4e688145f71d2dac31b92c623a0a67cefe6447c343f9c953ab6a0cc0
Jul 27 18:37:10.756363 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.756289    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv"]
Jul 27 18:37:10.759084 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.759067    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv"
Jul 27 18:37:10.770431 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.770407    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv"]
Jul 27 18:37:10.854204 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.854171    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b058ecf5-cd45-4a3d-bbc2-733a137142ec-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv\" (UID: \"b058ecf5-cd45-4a3d-bbc2-733a137142ec\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv"
Jul 27 18:37:10.854365 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.854314    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b058ecf5-cd45-4a3d-bbc2-733a137142ec-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv\" (UID: \"b058ecf5-cd45-4a3d-bbc2-733a137142ec\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv"
Jul 27 18:37:10.854365 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.854343    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz7bx\" (UniqueName: \"kubernetes.io/projected/b058ecf5-cd45-4a3d-bbc2-733a137142ec-kube-api-access-tz7bx\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv\" (UID: \"b058ecf5-cd45-4a3d-bbc2-733a137142ec\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv"
Jul 27 18:37:10.954909 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.954875    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b058ecf5-cd45-4a3d-bbc2-733a137142ec-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv\" (UID: \"b058ecf5-cd45-4a3d-bbc2-733a137142ec\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv"
Jul 27 18:37:10.954909 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.954910    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz7bx\" (UniqueName: \"kubernetes.io/projected/b058ecf5-cd45-4a3d-bbc2-733a137142ec-kube-api-access-tz7bx\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv\" (UID: \"b058ecf5-cd45-4a3d-bbc2-733a137142ec\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv"
Jul 27 18:37:10.955104 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.954934    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b058ecf5-cd45-4a3d-bbc2-733a137142ec-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv\" (UID: \"b058ecf5-cd45-4a3d-bbc2-733a137142ec\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv"
Jul 27 18:37:10.955285 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.955265    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b058ecf5-cd45-4a3d-bbc2-733a137142ec-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv\" (UID: \"b058ecf5-cd45-4a3d-bbc2-733a137142ec\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv"
Jul 27 18:37:10.955352 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.955331    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b058ecf5-cd45-4a3d-bbc2-733a137142ec-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv\" (UID: \"b058ecf5-cd45-4a3d-bbc2-733a137142ec\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv"
Jul 27 18:37:10.967447 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:10.964737    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz7bx\" (UniqueName: \"kubernetes.io/projected/b058ecf5-cd45-4a3d-bbc2-733a137142ec-kube-api-access-tz7bx\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv\" (UID: \"b058ecf5-cd45-4a3d-bbc2-733a137142ec\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv"
Jul 27 18:37:11.098588 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:11.098539    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv"
Jul 27 18:37:11.229870 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:11.229846    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv"]
Jul 27 18:37:11.231413 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:37:11.231391    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb058ecf5_cd45_4a3d_bbc2_733a137142ec.slice/crio-e47f2cd4607bd9c281cc403db46615f4ab1043275e8424b31b01759496e8d0d4 WatchSource:0}: Error finding container e47f2cd4607bd9c281cc403db46615f4ab1043275e8424b31b01759496e8d0d4: Status 404 returned error can't find the container with id e47f2cd4607bd9c281cc403db46615f4ab1043275e8424b31b01759496e8d0d4
Jul 27 18:37:11.359406 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:11.359331    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r"]
Jul 27 18:37:11.362191 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:11.362175    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r"
Jul 27 18:37:11.372550 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:11.372521    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r"]
Jul 27 18:37:11.396988 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:11.396952    2566 generic.go:356] "Generic (PLEG): container finished" podID="a0ccd514-6a2b-405f-a724-297b04ae55a7" containerID="f0d25e21b7308ba67e2efb10a538d06581e045371627cfb9ff43d4d225ea3e03" exitCode=0
Jul 27 18:37:11.397150 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:11.397038    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7" event={"ID":"a0ccd514-6a2b-405f-a724-297b04ae55a7","Type":"ContainerDied","Data":"f0d25e21b7308ba67e2efb10a538d06581e045371627cfb9ff43d4d225ea3e03"}
Jul 27 18:37:11.397212 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:11.397157    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7" event={"ID":"a0ccd514-6a2b-405f-a724-297b04ae55a7","Type":"ContainerStarted","Data":"6c25228f4e688145f71d2dac31b92c623a0a67cefe6447c343f9c953ab6a0cc0"}
Jul 27 18:37:11.399024 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:11.399000    2566 generic.go:356] "Generic (PLEG): container finished" podID="b058ecf5-cd45-4a3d-bbc2-733a137142ec" containerID="eb85c101d08caa69e4c7002b94f9eed7057dd36925588bdfd7b98de4c5467a81" exitCode=0
Jul 27 18:37:11.399156 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:11.399060    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv" event={"ID":"b058ecf5-cd45-4a3d-bbc2-733a137142ec","Type":"ContainerDied","Data":"eb85c101d08caa69e4c7002b94f9eed7057dd36925588bdfd7b98de4c5467a81"}
Jul 27 18:37:11.399156 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:11.399090    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv" event={"ID":"b058ecf5-cd45-4a3d-bbc2-733a137142ec","Type":"ContainerStarted","Data":"e47f2cd4607bd9c281cc403db46615f4ab1043275e8424b31b01759496e8d0d4"}
Jul 27 18:37:11.459844 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:11.459807    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3586023-5e0a-4e8a-b384-0d17e7337513-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r\" (UID: \"d3586023-5e0a-4e8a-b384-0d17e7337513\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r"
Jul 27 18:37:11.459844 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:11.459849    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3586023-5e0a-4e8a-b384-0d17e7337513-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r\" (UID: \"d3586023-5e0a-4e8a-b384-0d17e7337513\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r"
Jul 27 18:37:11.460072 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:11.459988    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4vms\" (UniqueName: \"kubernetes.io/projected/d3586023-5e0a-4e8a-b384-0d17e7337513-kube-api-access-p4vms\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r\" (UID: \"d3586023-5e0a-4e8a-b384-0d17e7337513\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r"
Jul 27 18:37:11.560596 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:11.560568    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3586023-5e0a-4e8a-b384-0d17e7337513-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r\" (UID: \"d3586023-5e0a-4e8a-b384-0d17e7337513\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r"
Jul 27 18:37:11.560596 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:11.560597    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3586023-5e0a-4e8a-b384-0d17e7337513-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r\" (UID: \"d3586023-5e0a-4e8a-b384-0d17e7337513\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r"
Jul 27 18:37:11.560790 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:11.560659    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4vms\" (UniqueName: \"kubernetes.io/projected/d3586023-5e0a-4e8a-b384-0d17e7337513-kube-api-access-p4vms\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r\" (UID: \"d3586023-5e0a-4e8a-b384-0d17e7337513\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r"
Jul 27 18:37:11.560993 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:11.560967    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3586023-5e0a-4e8a-b384-0d17e7337513-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r\" (UID: \"d3586023-5e0a-4e8a-b384-0d17e7337513\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r"
Jul 27 18:37:11.561030 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:11.560982    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3586023-5e0a-4e8a-b384-0d17e7337513-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r\" (UID: \"d3586023-5e0a-4e8a-b384-0d17e7337513\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r"
Jul 27 18:37:11.569138 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:11.569088    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4vms\" (UniqueName: \"kubernetes.io/projected/d3586023-5e0a-4e8a-b384-0d17e7337513-kube-api-access-p4vms\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r\" (UID: \"d3586023-5e0a-4e8a-b384-0d17e7337513\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r"
Jul 27 18:37:11.708217 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:11.708141    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r"
Jul 27 18:37:11.840971 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:11.840947    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r"]
Jul 27 18:37:11.842567 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:37:11.842540    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3586023_5e0a_4e8a_b384_0d17e7337513.slice/crio-9a1c95c78738e88c47c4b870a8c3a330bc7fa706de5ad7c4015161c46f358da3 WatchSource:0}: Error finding container 9a1c95c78738e88c47c4b870a8c3a330bc7fa706de5ad7c4015161c46f358da3: Status 404 returned error can't find the container with id 9a1c95c78738e88c47c4b870a8c3a330bc7fa706de5ad7c4015161c46f358da3
Jul 27 18:37:11.962577 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:11.962521    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx"]
Jul 27 18:37:11.965087 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:11.965071    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx"
Jul 27 18:37:11.976201 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:11.976175    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx"]
Jul 27 18:37:12.064969 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:12.064934    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjtmt\" (UniqueName: \"kubernetes.io/projected/9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a-kube-api-access-vjtmt\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx\" (UID: \"9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx"
Jul 27 18:37:12.064969 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:12.064974    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx\" (UID: \"9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx"
Jul 27 18:37:12.065284 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:12.064997    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx\" (UID: \"9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx"
Jul 27 18:37:12.166262 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:12.166232    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjtmt\" (UniqueName: \"kubernetes.io/projected/9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a-kube-api-access-vjtmt\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx\" (UID: \"9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx"
Jul 27 18:37:12.166403 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:12.166278    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx\" (UID: \"9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx"
Jul 27 18:37:12.166403 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:12.166310    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx\" (UID: \"9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx"
Jul 27 18:37:12.166691 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:12.166674    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx\" (UID: \"9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx"
Jul 27 18:37:12.166729 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:12.166711    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx\" (UID: \"9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx"
Jul 27 18:37:12.175484 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:12.175457    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjtmt\" (UniqueName: \"kubernetes.io/projected/9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a-kube-api-access-vjtmt\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx\" (UID: \"9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx"
Jul 27 18:37:12.315335 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:12.315311    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx"
Jul 27 18:37:12.408683 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:12.408651    2566 generic.go:356] "Generic (PLEG): container finished" podID="b058ecf5-cd45-4a3d-bbc2-733a137142ec" containerID="5a07c076141ed6fabcbd0f113289648f7aa4c21eb3da10d0c8415ba8097a842a" exitCode=0
Jul 27 18:37:12.408834 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:12.408740    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv" event={"ID":"b058ecf5-cd45-4a3d-bbc2-733a137142ec","Type":"ContainerDied","Data":"5a07c076141ed6fabcbd0f113289648f7aa4c21eb3da10d0c8415ba8097a842a"}
Jul 27 18:37:12.410704 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:12.410680    2566 generic.go:356] "Generic (PLEG): container finished" podID="d3586023-5e0a-4e8a-b384-0d17e7337513" containerID="f218ec425490a9d541a7e21432379c8108622b38516056c3abc50fd966be58cd" exitCode=0
Jul 27 18:37:12.410813 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:12.410750    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r" event={"ID":"d3586023-5e0a-4e8a-b384-0d17e7337513","Type":"ContainerDied","Data":"f218ec425490a9d541a7e21432379c8108622b38516056c3abc50fd966be58cd"}
Jul 27 18:37:12.410813 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:12.410795    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r" event={"ID":"d3586023-5e0a-4e8a-b384-0d17e7337513","Type":"ContainerStarted","Data":"9a1c95c78738e88c47c4b870a8c3a330bc7fa706de5ad7c4015161c46f358da3"}
Jul 27 18:37:12.451270 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:12.451245    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx"]
Jul 27 18:37:12.453156 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:37:12.453086    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d6de99c_69ea_4fcc_8fc2_8a2d23c3cf6a.slice/crio-22fd457d53b49a61d701db99df1e53ddf6a404aae95e3655d315ee8a9949e34b WatchSource:0}: Error finding container 22fd457d53b49a61d701db99df1e53ddf6a404aae95e3655d315ee8a9949e34b: Status 404 returned error can't find the container with id 22fd457d53b49a61d701db99df1e53ddf6a404aae95e3655d315ee8a9949e34b
Jul 27 18:37:12.609886 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:12.609859    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cb99d6978-pctpl"
Jul 27 18:37:12.610005 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:12.609932    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cb99d6978-pctpl"
Jul 27 18:37:12.614694 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:12.614674    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cb99d6978-pctpl"
Jul 27 18:37:13.417017 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:13.416986    2566 generic.go:356] "Generic (PLEG): container finished" podID="a0ccd514-6a2b-405f-a724-297b04ae55a7" containerID="b925bc493f202d8622ba51ccaf0feac1a450e6e9af4a527767255b1e166017f0" exitCode=0
Jul 27 18:37:13.417485 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:13.417075    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7" event={"ID":"a0ccd514-6a2b-405f-a724-297b04ae55a7","Type":"ContainerDied","Data":"b925bc493f202d8622ba51ccaf0feac1a450e6e9af4a527767255b1e166017f0"}
Jul 27 18:37:13.419292 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:13.419267    2566 generic.go:356] "Generic (PLEG): container finished" podID="b058ecf5-cd45-4a3d-bbc2-733a137142ec" containerID="24845123bb2ce2673004f57cd8656e6cf94bf076361df42dab606d23c46fe71c" exitCode=0
Jul 27 18:37:13.419371 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:13.419347    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv" event={"ID":"b058ecf5-cd45-4a3d-bbc2-733a137142ec","Type":"ContainerDied","Data":"24845123bb2ce2673004f57cd8656e6cf94bf076361df42dab606d23c46fe71c"}
Jul 27 18:37:13.420720 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:13.420696    2566 generic.go:356] "Generic (PLEG): container finished" podID="9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a" containerID="95bba445a9885c220f3da1aab05f0f9a155dd375f378e3041a83eea7583a9833" exitCode=0
Jul 27 18:37:13.420866 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:13.420817    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx" event={"ID":"9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a","Type":"ContainerDied","Data":"95bba445a9885c220f3da1aab05f0f9a155dd375f378e3041a83eea7583a9833"}
Jul 27 18:37:13.420982 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:13.420873    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx" event={"ID":"9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a","Type":"ContainerStarted","Data":"22fd457d53b49a61d701db99df1e53ddf6a404aae95e3655d315ee8a9949e34b"}
Jul 27 18:37:13.425299 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:13.425280    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cb99d6978-pctpl"
Jul 27 18:37:13.522489 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:13.522457    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58cf7c967-q62dt"]
Jul 27 18:37:14.429231 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:14.429194    2566 generic.go:356] "Generic (PLEG): container finished" podID="d3586023-5e0a-4e8a-b384-0d17e7337513" containerID="7f6ae3983d6c191ad67b6088ec0eab8b73ccc711ea2f4bb15cb33ba5804077b4" exitCode=0
Jul 27 18:37:14.429671 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:14.429291    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r" event={"ID":"d3586023-5e0a-4e8a-b384-0d17e7337513","Type":"ContainerDied","Data":"7f6ae3983d6c191ad67b6088ec0eab8b73ccc711ea2f4bb15cb33ba5804077b4"}
Jul 27 18:37:14.431464 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:14.431439    2566 generic.go:356] "Generic (PLEG): container finished" podID="a0ccd514-6a2b-405f-a724-297b04ae55a7" containerID="5e277d45317328b004012f92859bb42eb2f8340ece795c7f77bf44a4db1b7b9b" exitCode=0
Jul 27 18:37:14.431567 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:14.431468    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7" event={"ID":"a0ccd514-6a2b-405f-a724-297b04ae55a7","Type":"ContainerDied","Data":"5e277d45317328b004012f92859bb42eb2f8340ece795c7f77bf44a4db1b7b9b"}
Jul 27 18:37:14.672552 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:14.672529    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv"
Jul 27 18:37:14.790392 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:14.790366    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b058ecf5-cd45-4a3d-bbc2-733a137142ec-bundle\") pod \"b058ecf5-cd45-4a3d-bbc2-733a137142ec\" (UID: \"b058ecf5-cd45-4a3d-bbc2-733a137142ec\") "
Jul 27 18:37:14.790493 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:14.790471    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b058ecf5-cd45-4a3d-bbc2-733a137142ec-util\") pod \"b058ecf5-cd45-4a3d-bbc2-733a137142ec\" (UID: \"b058ecf5-cd45-4a3d-bbc2-733a137142ec\") "
Jul 27 18:37:14.790546 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:14.790534    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tz7bx\" (UniqueName: \"kubernetes.io/projected/b058ecf5-cd45-4a3d-bbc2-733a137142ec-kube-api-access-tz7bx\") pod \"b058ecf5-cd45-4a3d-bbc2-733a137142ec\" (UID: \"b058ecf5-cd45-4a3d-bbc2-733a137142ec\") "
Jul 27 18:37:14.791005 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:14.790979    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b058ecf5-cd45-4a3d-bbc2-733a137142ec-bundle" (OuterVolumeSpecName: "bundle") pod "b058ecf5-cd45-4a3d-bbc2-733a137142ec" (UID: "b058ecf5-cd45-4a3d-bbc2-733a137142ec"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:37:14.792651 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:14.792627    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b058ecf5-cd45-4a3d-bbc2-733a137142ec-kube-api-access-tz7bx" (OuterVolumeSpecName: "kube-api-access-tz7bx") pod "b058ecf5-cd45-4a3d-bbc2-733a137142ec" (UID: "b058ecf5-cd45-4a3d-bbc2-733a137142ec"). InnerVolumeSpecName "kube-api-access-tz7bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:37:14.795084 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:14.795065    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b058ecf5-cd45-4a3d-bbc2-733a137142ec-util" (OuterVolumeSpecName: "util") pod "b058ecf5-cd45-4a3d-bbc2-733a137142ec" (UID: "b058ecf5-cd45-4a3d-bbc2-733a137142ec"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:37:14.891531 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:14.891497    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b058ecf5-cd45-4a3d-bbc2-733a137142ec-util\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:37:14.891531 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:14.891526    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz7bx\" (UniqueName: \"kubernetes.io/projected/b058ecf5-cd45-4a3d-bbc2-733a137142ec-kube-api-access-tz7bx\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:37:14.891531 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:14.891537    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b058ecf5-cd45-4a3d-bbc2-733a137142ec-bundle\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:37:15.438428 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:15.438390    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv" event={"ID":"b058ecf5-cd45-4a3d-bbc2-733a137142ec","Type":"ContainerDied","Data":"e47f2cd4607bd9c281cc403db46615f4ab1043275e8424b31b01759496e8d0d4"}
Jul 27 18:37:15.438864 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:15.438449    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e47f2cd4607bd9c281cc403db46615f4ab1043275e8424b31b01759496e8d0d4"
Jul 27 18:37:15.438864 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:15.438402    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv"
Jul 27 18:37:15.440338 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:15.440316    2566 generic.go:356] "Generic (PLEG): container finished" podID="9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a" containerID="421f2c7a54771668e5fb86e24b613584cdd644688b70e1096b312091e361926d" exitCode=0
Jul 27 18:37:15.440455 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:15.440394    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx" event={"ID":"9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a","Type":"ContainerDied","Data":"421f2c7a54771668e5fb86e24b613584cdd644688b70e1096b312091e361926d"}
Jul 27 18:37:15.443319 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:15.443297    2566 generic.go:356] "Generic (PLEG): container finished" podID="d3586023-5e0a-4e8a-b384-0d17e7337513" containerID="d3e3489aad6f283dbb9792aeee5b96cd29d23ce1c8123944d9d69698f685f2fa" exitCode=0
Jul 27 18:37:15.443424 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:15.443364    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r" event={"ID":"d3586023-5e0a-4e8a-b384-0d17e7337513","Type":"ContainerDied","Data":"d3e3489aad6f283dbb9792aeee5b96cd29d23ce1c8123944d9d69698f685f2fa"}
Jul 27 18:37:15.635984 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:15.635961    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7"
Jul 27 18:37:15.697222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:15.697157    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0ccd514-6a2b-405f-a724-297b04ae55a7-util\") pod \"a0ccd514-6a2b-405f-a724-297b04ae55a7\" (UID: \"a0ccd514-6a2b-405f-a724-297b04ae55a7\") "
Jul 27 18:37:15.697222 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:15.697214    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzwcb\" (UniqueName: \"kubernetes.io/projected/a0ccd514-6a2b-405f-a724-297b04ae55a7-kube-api-access-gzwcb\") pod \"a0ccd514-6a2b-405f-a724-297b04ae55a7\" (UID: \"a0ccd514-6a2b-405f-a724-297b04ae55a7\") "
Jul 27 18:37:15.697393 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:15.697252    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0ccd514-6a2b-405f-a724-297b04ae55a7-bundle\") pod \"a0ccd514-6a2b-405f-a724-297b04ae55a7\" (UID: \"a0ccd514-6a2b-405f-a724-297b04ae55a7\") "
Jul 27 18:37:15.697845 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:15.697722    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ccd514-6a2b-405f-a724-297b04ae55a7-bundle" (OuterVolumeSpecName: "bundle") pod "a0ccd514-6a2b-405f-a724-297b04ae55a7" (UID: "a0ccd514-6a2b-405f-a724-297b04ae55a7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:37:15.699346 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:15.699323    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ccd514-6a2b-405f-a724-297b04ae55a7-kube-api-access-gzwcb" (OuterVolumeSpecName: "kube-api-access-gzwcb") pod "a0ccd514-6a2b-405f-a724-297b04ae55a7" (UID: "a0ccd514-6a2b-405f-a724-297b04ae55a7"). InnerVolumeSpecName "kube-api-access-gzwcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:37:15.702791 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:15.702758    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ccd514-6a2b-405f-a724-297b04ae55a7-util" (OuterVolumeSpecName: "util") pod "a0ccd514-6a2b-405f-a724-297b04ae55a7" (UID: "a0ccd514-6a2b-405f-a724-297b04ae55a7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:37:15.798439 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:15.798406    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0ccd514-6a2b-405f-a724-297b04ae55a7-util\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:37:15.798439 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:15.798433    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzwcb\" (UniqueName: \"kubernetes.io/projected/a0ccd514-6a2b-405f-a724-297b04ae55a7-kube-api-access-gzwcb\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:37:15.798439 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:15.798443    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0ccd514-6a2b-405f-a724-297b04ae55a7-bundle\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:37:16.451513 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:16.451430    2566 generic.go:356] "Generic (PLEG): container finished" podID="9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a" containerID="080ab9d522f83cf47155cb784c2f171c7e6bd6d883ec0623704636460496750b" exitCode=0
Jul 27 18:37:16.451879 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:16.451514    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx" event={"ID":"9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a","Type":"ContainerDied","Data":"080ab9d522f83cf47155cb784c2f171c7e6bd6d883ec0623704636460496750b"}
Jul 27 18:37:16.453538 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:16.453514    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7" event={"ID":"a0ccd514-6a2b-405f-a724-297b04ae55a7","Type":"ContainerDied","Data":"6c25228f4e688145f71d2dac31b92c623a0a67cefe6447c343f9c953ab6a0cc0"}
Jul 27 18:37:16.453645 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:16.453538    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7"
Jul 27 18:37:16.453645 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:16.453557    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c25228f4e688145f71d2dac31b92c623a0a67cefe6447c343f9c953ab6a0cc0"
Jul 27 18:37:16.589797 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:16.589774    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r"
Jul 27 18:37:16.705944 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:16.705865    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3586023-5e0a-4e8a-b384-0d17e7337513-util\") pod \"d3586023-5e0a-4e8a-b384-0d17e7337513\" (UID: \"d3586023-5e0a-4e8a-b384-0d17e7337513\") "
Jul 27 18:37:16.705944 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:16.705907    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4vms\" (UniqueName: \"kubernetes.io/projected/d3586023-5e0a-4e8a-b384-0d17e7337513-kube-api-access-p4vms\") pod \"d3586023-5e0a-4e8a-b384-0d17e7337513\" (UID: \"d3586023-5e0a-4e8a-b384-0d17e7337513\") "
Jul 27 18:37:16.705944 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:16.705938    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3586023-5e0a-4e8a-b384-0d17e7337513-bundle\") pod \"d3586023-5e0a-4e8a-b384-0d17e7337513\" (UID: \"d3586023-5e0a-4e8a-b384-0d17e7337513\") "
Jul 27 18:37:16.706467 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:16.706433    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3586023-5e0a-4e8a-b384-0d17e7337513-bundle" (OuterVolumeSpecName: "bundle") pod "d3586023-5e0a-4e8a-b384-0d17e7337513" (UID: "d3586023-5e0a-4e8a-b384-0d17e7337513"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:37:16.708131 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:16.708094    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3586023-5e0a-4e8a-b384-0d17e7337513-kube-api-access-p4vms" (OuterVolumeSpecName: "kube-api-access-p4vms") pod "d3586023-5e0a-4e8a-b384-0d17e7337513" (UID: "d3586023-5e0a-4e8a-b384-0d17e7337513"). InnerVolumeSpecName "kube-api-access-p4vms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:37:16.711085 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:16.711063    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3586023-5e0a-4e8a-b384-0d17e7337513-util" (OuterVolumeSpecName: "util") pod "d3586023-5e0a-4e8a-b384-0d17e7337513" (UID: "d3586023-5e0a-4e8a-b384-0d17e7337513"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:37:16.806634 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:16.806601    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3586023-5e0a-4e8a-b384-0d17e7337513-util\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:37:16.806634 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:16.806631    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4vms\" (UniqueName: \"kubernetes.io/projected/d3586023-5e0a-4e8a-b384-0d17e7337513-kube-api-access-p4vms\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:37:16.806801 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:16.806643    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3586023-5e0a-4e8a-b384-0d17e7337513-bundle\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:37:17.463388 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:17.463361    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r"
Jul 27 18:37:17.463843 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:17.463397    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r" event={"ID":"d3586023-5e0a-4e8a-b384-0d17e7337513","Type":"ContainerDied","Data":"9a1c95c78738e88c47c4b870a8c3a330bc7fa706de5ad7c4015161c46f358da3"}
Jul 27 18:37:17.463843 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:17.463446    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a1c95c78738e88c47c4b870a8c3a330bc7fa706de5ad7c4015161c46f358da3"
Jul 27 18:37:17.594415 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:17.594389    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx"
Jul 27 18:37:17.713907 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:17.713821    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjtmt\" (UniqueName: \"kubernetes.io/projected/9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a-kube-api-access-vjtmt\") pod \"9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a\" (UID: \"9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a\") "
Jul 27 18:37:17.713907 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:17.713894    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a-util\") pod \"9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a\" (UID: \"9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a\") "
Jul 27 18:37:17.714106 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:17.713936    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a-bundle\") pod \"9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a\" (UID: \"9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a\") "
Jul 27 18:37:17.714463 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:17.714422    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a-bundle" (OuterVolumeSpecName: "bundle") pod "9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a" (UID: "9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:37:17.716066 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:17.716047    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a-kube-api-access-vjtmt" (OuterVolumeSpecName: "kube-api-access-vjtmt") pod "9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a" (UID: "9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a"). InnerVolumeSpecName "kube-api-access-vjtmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:37:17.718350 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:17.718332    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a-util" (OuterVolumeSpecName: "util") pod "9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a" (UID: "9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:37:17.815101 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:17.815066    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a-util\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:37:17.815101 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:17.815094    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a-bundle\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:37:17.815101 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:17.815103    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjtmt\" (UniqueName: \"kubernetes.io/projected/9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a-kube-api-access-vjtmt\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:37:18.470813 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:18.470735    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx" event={"ID":"9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a","Type":"ContainerDied","Data":"22fd457d53b49a61d701db99df1e53ddf6a404aae95e3655d315ee8a9949e34b"}
Jul 27 18:37:18.470813 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:18.470780    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22fd457d53b49a61d701db99df1e53ddf6a404aae95e3655d315ee8a9949e34b"
Jul 27 18:37:18.470813 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:18.470780    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx"
Jul 27 18:37:38.549526 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:38.549467    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-58cf7c967-q62dt" podUID="b3023d8c-9812-4e80-93a4-e8f4ee0ab736" containerName="console" containerID="cri-o://b22c70b13e70b9df05484db97e1c6983fe7b42f49a07d6cf539a9ce4a6ad4383" gracePeriod=15
Jul 27 18:37:38.801646 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:38.801581    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:37:38.890663 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:38.890626    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-service-ca\") pod \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") "
Jul 27 18:37:38.890833 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:38.890694    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-console-oauth-config\") pod \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") "
Jul 27 18:37:38.890833 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:38.890737    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-console-serving-cert\") pod \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") "
Jul 27 18:37:38.890833 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:38.890796    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pvvs\" (UniqueName: \"kubernetes.io/projected/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-kube-api-access-6pvvs\") pod \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") "
Jul 27 18:37:38.890833 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:38.890829    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-trusted-ca-bundle\") pod \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") "
Jul 27 18:37:38.891024 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:38.890874    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-console-config\") pod \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") "
Jul 27 18:37:38.891024 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:38.890898    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-oauth-serving-cert\") pod \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\" (UID: \"b3023d8c-9812-4e80-93a4-e8f4ee0ab736\") "
Jul 27 18:37:38.891152 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:38.891054    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-service-ca" (OuterVolumeSpecName: "service-ca") pod "b3023d8c-9812-4e80-93a4-e8f4ee0ab736" (UID: "b3023d8c-9812-4e80-93a4-e8f4ee0ab736"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:37:38.891402 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:38.891363    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-service-ca\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:37:38.891402 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:38.891381    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-console-config" (OuterVolumeSpecName: "console-config") pod "b3023d8c-9812-4e80-93a4-e8f4ee0ab736" (UID: "b3023d8c-9812-4e80-93a4-e8f4ee0ab736"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:37:38.891712 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:38.891423    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b3023d8c-9812-4e80-93a4-e8f4ee0ab736" (UID: "b3023d8c-9812-4e80-93a4-e8f4ee0ab736"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:37:38.891712 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:38.891433    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b3023d8c-9812-4e80-93a4-e8f4ee0ab736" (UID: "b3023d8c-9812-4e80-93a4-e8f4ee0ab736"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:37:38.893090 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:38.893064    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b3023d8c-9812-4e80-93a4-e8f4ee0ab736" (UID: "b3023d8c-9812-4e80-93a4-e8f4ee0ab736"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:37:38.893490 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:38.893469    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b3023d8c-9812-4e80-93a4-e8f4ee0ab736" (UID: "b3023d8c-9812-4e80-93a4-e8f4ee0ab736"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:37:38.893583 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:38.893563    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-kube-api-access-6pvvs" (OuterVolumeSpecName: "kube-api-access-6pvvs") pod "b3023d8c-9812-4e80-93a4-e8f4ee0ab736" (UID: "b3023d8c-9812-4e80-93a4-e8f4ee0ab736"). InnerVolumeSpecName "kube-api-access-6pvvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:37:38.991830 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:38.991806    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-console-oauth-config\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:37:38.991830 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:38.991825    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-console-serving-cert\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:37:38.991830 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:38.991834    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pvvs\" (UniqueName: \"kubernetes.io/projected/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-kube-api-access-6pvvs\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:37:38.991995 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:38.991843    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-trusted-ca-bundle\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:37:38.991995 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:38.991851    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-console-config\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:37:38.991995 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:38.991859    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3023d8c-9812-4e80-93a4-e8f4ee0ab736-oauth-serving-cert\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:37:39.584433 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:39.584397    2566 generic.go:356] "Generic (PLEG): container finished" podID="b3023d8c-9812-4e80-93a4-e8f4ee0ab736" containerID="b22c70b13e70b9df05484db97e1c6983fe7b42f49a07d6cf539a9ce4a6ad4383" exitCode=0
Jul 27 18:37:39.584841 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:39.584446    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58cf7c967-q62dt"
Jul 27 18:37:39.584841 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:39.584486    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58cf7c967-q62dt" event={"ID":"b3023d8c-9812-4e80-93a4-e8f4ee0ab736","Type":"ContainerDied","Data":"b22c70b13e70b9df05484db97e1c6983fe7b42f49a07d6cf539a9ce4a6ad4383"}
Jul 27 18:37:39.584841 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:39.584550    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58cf7c967-q62dt" event={"ID":"b3023d8c-9812-4e80-93a4-e8f4ee0ab736","Type":"ContainerDied","Data":"104054455f8c152e4b5e3cc3a71613247a8a0e648ba5e048b1e2cae21b7eb716"}
Jul 27 18:37:39.584841 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:39.584569    2566 scope.go:117] "RemoveContainer" containerID="b22c70b13e70b9df05484db97e1c6983fe7b42f49a07d6cf539a9ce4a6ad4383"
Jul 27 18:37:39.593815 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:39.593802    2566 scope.go:117] "RemoveContainer" containerID="b22c70b13e70b9df05484db97e1c6983fe7b42f49a07d6cf539a9ce4a6ad4383"
Jul 27 18:37:39.594071 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:37:39.594054    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b22c70b13e70b9df05484db97e1c6983fe7b42f49a07d6cf539a9ce4a6ad4383\": container with ID starting with b22c70b13e70b9df05484db97e1c6983fe7b42f49a07d6cf539a9ce4a6ad4383 not found: ID does not exist" containerID="b22c70b13e70b9df05484db97e1c6983fe7b42f49a07d6cf539a9ce4a6ad4383"
Jul 27 18:37:39.594144 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:39.594079    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b22c70b13e70b9df05484db97e1c6983fe7b42f49a07d6cf539a9ce4a6ad4383"} err="failed to get container status \"b22c70b13e70b9df05484db97e1c6983fe7b42f49a07d6cf539a9ce4a6ad4383\": rpc error: code = NotFound desc = could not find container \"b22c70b13e70b9df05484db97e1c6983fe7b42f49a07d6cf539a9ce4a6ad4383\": container with ID starting with b22c70b13e70b9df05484db97e1c6983fe7b42f49a07d6cf539a9ce4a6ad4383 not found: ID does not exist"
Jul 27 18:37:39.611590 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:39.611570    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58cf7c967-q62dt"]
Jul 27 18:37:39.621750 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:39.621728    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58cf7c967-q62dt"]
Jul 27 18:37:40.123508 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:40.123470    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3023d8c-9812-4e80-93a4-e8f4ee0ab736" path="/var/lib/kubelet/pods/b3023d8c-9812-4e80-93a4-e8f4ee0ab736/volumes"
Jul 27 18:37:56.840004 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.839969    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m84gm"]
Jul 27 18:37:56.840463 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840447    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ccd514-6a2b-405f-a724-297b04ae55a7" containerName="extract"
Jul 27 18:37:56.840505 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840465    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ccd514-6a2b-405f-a724-297b04ae55a7" containerName="extract"
Jul 27 18:37:56.840505 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840476    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3586023-5e0a-4e8a-b384-0d17e7337513" containerName="extract"
Jul 27 18:37:56.840505 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840481    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3586023-5e0a-4e8a-b384-0d17e7337513" containerName="extract"
Jul 27 18:37:56.840505 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840490    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b058ecf5-cd45-4a3d-bbc2-733a137142ec" containerName="extract"
Jul 27 18:37:56.840505 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840495    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b058ecf5-cd45-4a3d-bbc2-733a137142ec" containerName="extract"
Jul 27 18:37:56.840505 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840503    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ccd514-6a2b-405f-a724-297b04ae55a7" containerName="util"
Jul 27 18:37:56.840680 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840509    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ccd514-6a2b-405f-a724-297b04ae55a7" containerName="util"
Jul 27 18:37:56.840680 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840514    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a" containerName="util"
Jul 27 18:37:56.840680 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840519    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a" containerName="util"
Jul 27 18:37:56.840680 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840525    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b058ecf5-cd45-4a3d-bbc2-733a137142ec" containerName="pull"
Jul 27 18:37:56.840680 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840530    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b058ecf5-cd45-4a3d-bbc2-733a137142ec" containerName="pull"
Jul 27 18:37:56.840680 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840537    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3586023-5e0a-4e8a-b384-0d17e7337513" containerName="pull"
Jul 27 18:37:56.840680 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840542    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3586023-5e0a-4e8a-b384-0d17e7337513" containerName="pull"
Jul 27 18:37:56.840680 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840548    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ccd514-6a2b-405f-a724-297b04ae55a7" containerName="pull"
Jul 27 18:37:56.840680 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840552    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ccd514-6a2b-405f-a724-297b04ae55a7" containerName="pull"
Jul 27 18:37:56.840680 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840558    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a" containerName="pull"
Jul 27 18:37:56.840680 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840562    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a" containerName="pull"
Jul 27 18:37:56.840680 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840568    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a" containerName="extract"
Jul 27 18:37:56.840680 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840572    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a" containerName="extract"
Jul 27 18:37:56.840680 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840578    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3586023-5e0a-4e8a-b384-0d17e7337513" containerName="util"
Jul 27 18:37:56.840680 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840582    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3586023-5e0a-4e8a-b384-0d17e7337513" containerName="util"
Jul 27 18:37:56.840680 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840592    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3023d8c-9812-4e80-93a4-e8f4ee0ab736" containerName="console"
Jul 27 18:37:56.840680 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840596    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3023d8c-9812-4e80-93a4-e8f4ee0ab736" containerName="console"
Jul 27 18:37:56.840680 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840601    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b058ecf5-cd45-4a3d-bbc2-733a137142ec" containerName="util"
Jul 27 18:37:56.840680 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840606    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b058ecf5-cd45-4a3d-bbc2-733a137142ec" containerName="util"
Jul 27 18:37:56.840680 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840658    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3023d8c-9812-4e80-93a4-e8f4ee0ab736" containerName="console"
Jul 27 18:37:56.840680 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840664    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0ccd514-6a2b-405f-a724-297b04ae55a7" containerName="extract"
Jul 27 18:37:56.840680 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840671    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a" containerName="extract"
Jul 27 18:37:56.840680 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840676    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b058ecf5-cd45-4a3d-bbc2-733a137142ec" containerName="extract"
Jul 27 18:37:56.840680 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.840681    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3586023-5e0a-4e8a-b384-0d17e7337513" containerName="extract"
Jul 27 18:37:56.843731 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.843715    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m84gm"
Jul 27 18:37:56.846906 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.846889    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 27 18:37:56.848196 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.848180    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 27 18:37:56.848196 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.848190    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-2n2bn\""
Jul 27 18:37:56.855086 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.855062    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m84gm"]
Jul 27 18:37:56.938800 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:56.938765    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znqng\" (UniqueName: \"kubernetes.io/projected/972972f9-1bef-4c24-8b1a-0c6076d73dab-kube-api-access-znqng\") pod \"limitador-operator-controller-manager-859d7b78bd-m84gm\" (UID: \"972972f9-1bef-4c24-8b1a-0c6076d73dab\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m84gm"
Jul 27 18:37:57.040028 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:57.039998    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znqng\" (UniqueName: \"kubernetes.io/projected/972972f9-1bef-4c24-8b1a-0c6076d73dab-kube-api-access-znqng\") pod \"limitador-operator-controller-manager-859d7b78bd-m84gm\" (UID: \"972972f9-1bef-4c24-8b1a-0c6076d73dab\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m84gm"
Jul 27 18:37:57.050020 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:57.049996    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znqng\" (UniqueName: \"kubernetes.io/projected/972972f9-1bef-4c24-8b1a-0c6076d73dab-kube-api-access-znqng\") pod \"limitador-operator-controller-manager-859d7b78bd-m84gm\" (UID: \"972972f9-1bef-4c24-8b1a-0c6076d73dab\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m84gm"
Jul 27 18:37:57.155501 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:57.155428    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m84gm"
Jul 27 18:37:57.293030 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:57.292991    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m84gm"]
Jul 27 18:37:57.299015 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:37:57.298971    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod972972f9_1bef_4c24_8b1a_0c6076d73dab.slice/crio-5f7541b15f4d986934d38aba7f18bdc13a8e900f2ca6351b419777f560823829 WatchSource:0}: Error finding container 5f7541b15f4d986934d38aba7f18bdc13a8e900f2ca6351b419777f560823829: Status 404 returned error can't find the container with id 5f7541b15f4d986934d38aba7f18bdc13a8e900f2ca6351b419777f560823829
Jul 27 18:37:57.675985 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:37:57.675948    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m84gm" event={"ID":"972972f9-1bef-4c24-8b1a-0c6076d73dab","Type":"ContainerStarted","Data":"5f7541b15f4d986934d38aba7f18bdc13a8e900f2ca6351b419777f560823829"}
Jul 27 18:38:00.694992 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:38:00.694908    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m84gm" event={"ID":"972972f9-1bef-4c24-8b1a-0c6076d73dab","Type":"ContainerStarted","Data":"ab1fc80e088400e13d746443eb5eaa2559305a07267d7ad98de2390e0b37affc"}
Jul 27 18:38:00.695371 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:38:00.695040    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m84gm"
Jul 27 18:38:00.718333 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:38:00.718281    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m84gm" podStartSLOduration=1.797833866 podStartE2EDuration="4.718265081s" totalImagesPullingTime="2.920431215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:38:00.716320166 +0000 UTC m=+541.150577788" watchObservedRunningTime="2026-07-27 18:38:00.718265081 +0000 UTC m=+541.152522692"
Jul 27 18:38:11.702255 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:38:11.702219    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m84gm"
Jul 27 18:38:53.761320 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:38:53.761286    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-mhcgz"]
Jul 27 18:38:53.764761 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:38:53.764745    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-mhcgz"
Jul 27 18:38:53.767686 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:38:53.767667    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-d825p\""
Jul 27 18:38:53.772815 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:38:53.772792    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-mhcgz"]
Jul 27 18:38:53.826272 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:38:53.826246    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxzs4\" (UniqueName: \"kubernetes.io/projected/30fd7696-ddf4-4a1e-9df0-54bd49100ffe-kube-api-access-hxzs4\") pod \"authorino-75d556bd77-mhcgz\" (UID: \"30fd7696-ddf4-4a1e-9df0-54bd49100ffe\") " pod="kuadrant-system/authorino-75d556bd77-mhcgz"
Jul 27 18:38:53.927344 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:38:53.927313    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxzs4\" (UniqueName: \"kubernetes.io/projected/30fd7696-ddf4-4a1e-9df0-54bd49100ffe-kube-api-access-hxzs4\") pod \"authorino-75d556bd77-mhcgz\" (UID: \"30fd7696-ddf4-4a1e-9df0-54bd49100ffe\") " pod="kuadrant-system/authorino-75d556bd77-mhcgz"
Jul 27 18:38:53.935989 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:38:53.935961    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxzs4\" (UniqueName: \"kubernetes.io/projected/30fd7696-ddf4-4a1e-9df0-54bd49100ffe-kube-api-access-hxzs4\") pod \"authorino-75d556bd77-mhcgz\" (UID: \"30fd7696-ddf4-4a1e-9df0-54bd49100ffe\") " pod="kuadrant-system/authorino-75d556bd77-mhcgz"
Jul 27 18:38:54.074990 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:38:54.074966    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-mhcgz"
Jul 27 18:38:54.233407 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:38:54.233377    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30fd7696_ddf4_4a1e_9df0_54bd49100ffe.slice/crio-9efd38446c177ea21300e0c2bf4860eb8201793adebc2f845be9d525f3216514 WatchSource:0}: Error finding container 9efd38446c177ea21300e0c2bf4860eb8201793adebc2f845be9d525f3216514: Status 404 returned error can't find the container with id 9efd38446c177ea21300e0c2bf4860eb8201793adebc2f845be9d525f3216514
Jul 27 18:38:54.236208 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:38:54.236182    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-mhcgz"]
Jul 27 18:38:54.968046 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:38:54.968014    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-mhcgz" event={"ID":"30fd7696-ddf4-4a1e-9df0-54bd49100ffe","Type":"ContainerStarted","Data":"9efd38446c177ea21300e0c2bf4860eb8201793adebc2f845be9d525f3216514"}
Jul 27 18:38:58.992716 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:38:58.992684    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-mhcgz" event={"ID":"30fd7696-ddf4-4a1e-9df0-54bd49100ffe","Type":"ContainerStarted","Data":"db87389ad55bd1a19b563f9efbc9385d6f5f3a6fe8b817c6fcfdc587797d068c"}
Jul 27 18:38:59.010033 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:38:59.009985    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-mhcgz" podStartSLOduration=2.276356689 podStartE2EDuration="6.009968934s" totalImagesPullingTime="3.733612245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:38:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:38:59.008625917 +0000 UTC m=+599.442883541" watchObservedRunningTime="2026-07-27 18:38:59.009968934 +0000 UTC m=+599.444226556"
Jul 27 18:39:00.005782 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:00.005754    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 18:39:00.010878 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:00.010858    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 18:39:26.504588 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:26.504512    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-mhcgz"]
Jul 27 18:39:26.507191 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:26.504724    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-mhcgz" podUID="30fd7696-ddf4-4a1e-9df0-54bd49100ffe" containerName="authorino" containerID="cri-o://db87389ad55bd1a19b563f9efbc9385d6f5f3a6fe8b817c6fcfdc587797d068c" gracePeriod=30
Jul 27 18:39:26.755431 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:26.755379    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-mhcgz"
Jul 27 18:39:26.916576 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:26.916538    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxzs4\" (UniqueName: \"kubernetes.io/projected/30fd7696-ddf4-4a1e-9df0-54bd49100ffe-kube-api-access-hxzs4\") pod \"30fd7696-ddf4-4a1e-9df0-54bd49100ffe\" (UID: \"30fd7696-ddf4-4a1e-9df0-54bd49100ffe\") "
Jul 27 18:39:26.918689 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:26.918658    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30fd7696-ddf4-4a1e-9df0-54bd49100ffe-kube-api-access-hxzs4" (OuterVolumeSpecName: "kube-api-access-hxzs4") pod "30fd7696-ddf4-4a1e-9df0-54bd49100ffe" (UID: "30fd7696-ddf4-4a1e-9df0-54bd49100ffe"). InnerVolumeSpecName "kube-api-access-hxzs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:39:27.017751 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:27.017693    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxzs4\" (UniqueName: \"kubernetes.io/projected/30fd7696-ddf4-4a1e-9df0-54bd49100ffe-kube-api-access-hxzs4\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:39:27.140779 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:27.140748    2566 generic.go:356] "Generic (PLEG): container finished" podID="30fd7696-ddf4-4a1e-9df0-54bd49100ffe" containerID="db87389ad55bd1a19b563f9efbc9385d6f5f3a6fe8b817c6fcfdc587797d068c" exitCode=0
Jul 27 18:39:27.140900 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:27.140787    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-mhcgz"
Jul 27 18:39:27.140900 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:27.140832    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-mhcgz" event={"ID":"30fd7696-ddf4-4a1e-9df0-54bd49100ffe","Type":"ContainerDied","Data":"db87389ad55bd1a19b563f9efbc9385d6f5f3a6fe8b817c6fcfdc587797d068c"}
Jul 27 18:39:27.140900 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:27.140866    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-mhcgz" event={"ID":"30fd7696-ddf4-4a1e-9df0-54bd49100ffe","Type":"ContainerDied","Data":"9efd38446c177ea21300e0c2bf4860eb8201793adebc2f845be9d525f3216514"}
Jul 27 18:39:27.140900 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:27.140883    2566 scope.go:117] "RemoveContainer" containerID="db87389ad55bd1a19b563f9efbc9385d6f5f3a6fe8b817c6fcfdc587797d068c"
Jul 27 18:39:27.150273 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:27.150257    2566 scope.go:117] "RemoveContainer" containerID="db87389ad55bd1a19b563f9efbc9385d6f5f3a6fe8b817c6fcfdc587797d068c"
Jul 27 18:39:27.150541 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:39:27.150520    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db87389ad55bd1a19b563f9efbc9385d6f5f3a6fe8b817c6fcfdc587797d068c\": container with ID starting with db87389ad55bd1a19b563f9efbc9385d6f5f3a6fe8b817c6fcfdc587797d068c not found: ID does not exist" containerID="db87389ad55bd1a19b563f9efbc9385d6f5f3a6fe8b817c6fcfdc587797d068c"
Jul 27 18:39:27.150604 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:27.150551    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db87389ad55bd1a19b563f9efbc9385d6f5f3a6fe8b817c6fcfdc587797d068c"} err="failed to get container status \"db87389ad55bd1a19b563f9efbc9385d6f5f3a6fe8b817c6fcfdc587797d068c\": rpc error: code = NotFound desc = could not find container \"db87389ad55bd1a19b563f9efbc9385d6f5f3a6fe8b817c6fcfdc587797d068c\": container with ID starting with db87389ad55bd1a19b563f9efbc9385d6f5f3a6fe8b817c6fcfdc587797d068c not found: ID does not exist"
Jul 27 18:39:27.163491 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:27.163471    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-mhcgz"]
Jul 27 18:39:27.169592 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:27.169574    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-mhcgz"]
Jul 27 18:39:28.123557 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:28.123520    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30fd7696-ddf4-4a1e-9df0-54bd49100ffe" path="/var/lib/kubelet/pods/30fd7696-ddf4-4a1e-9df0-54bd49100ffe/volumes"
Jul 27 18:39:31.812624 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:31.812590    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77ff6dccdd-424r7"]
Jul 27 18:39:31.813076 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:31.812963    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30fd7696-ddf4-4a1e-9df0-54bd49100ffe" containerName="authorino"
Jul 27 18:39:31.813076 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:31.812974    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="30fd7696-ddf4-4a1e-9df0-54bd49100ffe" containerName="authorino"
Jul 27 18:39:31.813076 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:31.813033    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="30fd7696-ddf4-4a1e-9df0-54bd49100ffe" containerName="authorino"
Jul 27 18:39:31.817294 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:31.817276    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77ff6dccdd-424r7"
Jul 27 18:39:31.820253 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:31.820233    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 27 18:39:31.820351 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:31.820290    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-chdqz\""
Jul 27 18:39:31.820351 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:31.820290    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 27 18:39:31.830486 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:31.830466    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77ff6dccdd-424r7"]
Jul 27 18:39:31.855462 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:31.855435    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47855e8a-9217-45dc-83fa-830c9928128e-webhook-cert\") pod \"maas-controller-77ff6dccdd-424r7\" (UID: \"47855e8a-9217-45dc-83fa-830c9928128e\") " pod="opendatahub/maas-controller-77ff6dccdd-424r7"
Jul 27 18:39:31.855570 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:31.855469    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk92t\" (UniqueName: \"kubernetes.io/projected/47855e8a-9217-45dc-83fa-830c9928128e-kube-api-access-lk92t\") pod \"maas-controller-77ff6dccdd-424r7\" (UID: \"47855e8a-9217-45dc-83fa-830c9928128e\") " pod="opendatahub/maas-controller-77ff6dccdd-424r7"
Jul 27 18:39:31.956618 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:31.956589    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47855e8a-9217-45dc-83fa-830c9928128e-webhook-cert\") pod \"maas-controller-77ff6dccdd-424r7\" (UID: \"47855e8a-9217-45dc-83fa-830c9928128e\") " pod="opendatahub/maas-controller-77ff6dccdd-424r7"
Jul 27 18:39:31.956778 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:31.956626    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk92t\" (UniqueName: \"kubernetes.io/projected/47855e8a-9217-45dc-83fa-830c9928128e-kube-api-access-lk92t\") pod \"maas-controller-77ff6dccdd-424r7\" (UID: \"47855e8a-9217-45dc-83fa-830c9928128e\") " pod="opendatahub/maas-controller-77ff6dccdd-424r7"
Jul 27 18:39:31.959283 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:31.959258    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47855e8a-9217-45dc-83fa-830c9928128e-webhook-cert\") pod \"maas-controller-77ff6dccdd-424r7\" (UID: \"47855e8a-9217-45dc-83fa-830c9928128e\") " pod="opendatahub/maas-controller-77ff6dccdd-424r7"
Jul 27 18:39:31.965827 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:31.965787    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk92t\" (UniqueName: \"kubernetes.io/projected/47855e8a-9217-45dc-83fa-830c9928128e-kube-api-access-lk92t\") pod \"maas-controller-77ff6dccdd-424r7\" (UID: \"47855e8a-9217-45dc-83fa-830c9928128e\") " pod="opendatahub/maas-controller-77ff6dccdd-424r7"
Jul 27 18:39:32.127562 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:32.127491    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77ff6dccdd-424r7"
Jul 27 18:39:32.264041 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:32.264009    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77ff6dccdd-424r7"]
Jul 27 18:39:32.265678 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:39:32.265650    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47855e8a_9217_45dc_83fa_830c9928128e.slice/crio-de8b26fbe1fe6fcb68cafd1a29346530c01c610841b2872e8890b4297e756363 WatchSource:0}: Error finding container de8b26fbe1fe6fcb68cafd1a29346530c01c610841b2872e8890b4297e756363: Status 404 returned error can't find the container with id de8b26fbe1fe6fcb68cafd1a29346530c01c610841b2872e8890b4297e756363
Jul 27 18:39:32.267247 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:32.267229    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 18:39:33.174724 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:33.174686    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77ff6dccdd-424r7" event={"ID":"47855e8a-9217-45dc-83fa-830c9928128e","Type":"ContainerStarted","Data":"de8b26fbe1fe6fcb68cafd1a29346530c01c610841b2872e8890b4297e756363"}
Jul 27 18:39:35.189036 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:35.188999    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77ff6dccdd-424r7" event={"ID":"47855e8a-9217-45dc-83fa-830c9928128e","Type":"ContainerStarted","Data":"de2c7608e3a51bcdb0d377b72ef10058743e9e90933c58841ed7a7ff303ebc35"}
Jul 27 18:39:35.189452 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:35.189081    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77ff6dccdd-424r7"
Jul 27 18:39:35.210845 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:35.210799    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-77ff6dccdd-424r7" podStartSLOduration=1.973574975 podStartE2EDuration="4.210785095s" totalImagesPullingTime="2.23721012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:39:35.209098888 +0000 UTC m=+635.643356501" watchObservedRunningTime="2026-07-27 18:39:35.210785095 +0000 UTC m=+635.645042710"
Jul 27 18:39:46.196973 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:39:46.196945    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77ff6dccdd-424r7"
Jul 27 18:40:07.072048 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:07.071964    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-557776f786-xvfwc"]
Jul 27 18:40:07.075252 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:07.075229    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-557776f786-xvfwc"
Jul 27 18:40:07.081706 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:07.081516    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 27 18:40:07.081706 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:07.081544    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 18:40:07.081835 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:07.081751    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 18:40:07.108462 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:07.108436    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-557776f786-xvfwc"]
Jul 27 18:40:07.163942 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:07.163904    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb9mz\" (UniqueName: \"kubernetes.io/projected/af0c2f08-b3af-4f51-944b-22262f3f7e18-kube-api-access-kb9mz\") pod \"maas-api-557776f786-xvfwc\" (UID: \"af0c2f08-b3af-4f51-944b-22262f3f7e18\") " pod="odh-ai-gateway-infra/maas-api-557776f786-xvfwc"
Jul 27 18:40:07.164183 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:07.163946    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af0c2f08-b3af-4f51-944b-22262f3f7e18-openshift-service-ca\") pod \"maas-api-557776f786-xvfwc\" (UID: \"af0c2f08-b3af-4f51-944b-22262f3f7e18\") " pod="odh-ai-gateway-infra/maas-api-557776f786-xvfwc"
Jul 27 18:40:07.164183 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:07.163996    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af0c2f08-b3af-4f51-944b-22262f3f7e18-maas-api-tls\") pod \"maas-api-557776f786-xvfwc\" (UID: \"af0c2f08-b3af-4f51-944b-22262f3f7e18\") " pod="odh-ai-gateway-infra/maas-api-557776f786-xvfwc"
Jul 27 18:40:07.264856 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:07.264819    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb9mz\" (UniqueName: \"kubernetes.io/projected/af0c2f08-b3af-4f51-944b-22262f3f7e18-kube-api-access-kb9mz\") pod \"maas-api-557776f786-xvfwc\" (UID: \"af0c2f08-b3af-4f51-944b-22262f3f7e18\") " pod="odh-ai-gateway-infra/maas-api-557776f786-xvfwc"
Jul 27 18:40:07.265048 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:07.264869    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af0c2f08-b3af-4f51-944b-22262f3f7e18-openshift-service-ca\") pod \"maas-api-557776f786-xvfwc\" (UID: \"af0c2f08-b3af-4f51-944b-22262f3f7e18\") " pod="odh-ai-gateway-infra/maas-api-557776f786-xvfwc"
Jul 27 18:40:07.265048 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:07.264917    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af0c2f08-b3af-4f51-944b-22262f3f7e18-maas-api-tls\") pod \"maas-api-557776f786-xvfwc\" (UID: \"af0c2f08-b3af-4f51-944b-22262f3f7e18\") " pod="odh-ai-gateway-infra/maas-api-557776f786-xvfwc"
Jul 27 18:40:07.265725 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:07.265699    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af0c2f08-b3af-4f51-944b-22262f3f7e18-openshift-service-ca\") pod \"maas-api-557776f786-xvfwc\" (UID: \"af0c2f08-b3af-4f51-944b-22262f3f7e18\") " pod="odh-ai-gateway-infra/maas-api-557776f786-xvfwc"
Jul 27 18:40:07.267603 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:07.267581    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af0c2f08-b3af-4f51-944b-22262f3f7e18-maas-api-tls\") pod \"maas-api-557776f786-xvfwc\" (UID: \"af0c2f08-b3af-4f51-944b-22262f3f7e18\") " pod="odh-ai-gateway-infra/maas-api-557776f786-xvfwc"
Jul 27 18:40:07.287450 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:07.287428    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb9mz\" (UniqueName: \"kubernetes.io/projected/af0c2f08-b3af-4f51-944b-22262f3f7e18-kube-api-access-kb9mz\") pod \"maas-api-557776f786-xvfwc\" (UID: \"af0c2f08-b3af-4f51-944b-22262f3f7e18\") " pod="odh-ai-gateway-infra/maas-api-557776f786-xvfwc"
Jul 27 18:40:07.385598 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:07.385518    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-557776f786-xvfwc"
Jul 27 18:40:07.533587 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:07.533507    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-557776f786-xvfwc"]
Jul 27 18:40:07.538244 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:40:07.538216    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf0c2f08_b3af_4f51_944b_22262f3f7e18.slice/crio-ecf50efb4f6ede20aa5d04719b68e14f7c05c28045b6caca61e990c50ab084d0 WatchSource:0}: Error finding container ecf50efb4f6ede20aa5d04719b68e14f7c05c28045b6caca61e990c50ab084d0: Status 404 returned error can't find the container with id ecf50efb4f6ede20aa5d04719b68e14f7c05c28045b6caca61e990c50ab084d0
Jul 27 18:40:08.008190 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:08.008155    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-4jtjz"]
Jul 27 18:40:08.037033 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:08.037002    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-4jtjz"]
Jul 27 18:40:08.037214 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:08.037044    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-4jtjz"
Jul 27 18:40:08.045320 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:08.045302    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 27 18:40:08.045668 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:08.045654    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-m457z\""
Jul 27 18:40:08.174852 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:08.174813    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebee721b-f5ea-448d-adf8-5cf2398fa566-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-4jtjz\" (UID: \"ebee721b-f5ea-448d-adf8-5cf2398fa566\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-4jtjz"
Jul 27 18:40:08.175332 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:08.174870    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvkrq\" (UniqueName: \"kubernetes.io/projected/ebee721b-f5ea-448d-adf8-5cf2398fa566-kube-api-access-wvkrq\") pod \"payload-pre-processing-687dbb6cf6-4jtjz\" (UID: \"ebee721b-f5ea-448d-adf8-5cf2398fa566\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-4jtjz"
Jul 27 18:40:08.275440 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:08.275409    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebee721b-f5ea-448d-adf8-5cf2398fa566-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-4jtjz\" (UID: \"ebee721b-f5ea-448d-adf8-5cf2398fa566\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-4jtjz"
Jul 27 18:40:08.275598 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:08.275480    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvkrq\" (UniqueName: \"kubernetes.io/projected/ebee721b-f5ea-448d-adf8-5cf2398fa566-kube-api-access-wvkrq\") pod \"payload-pre-processing-687dbb6cf6-4jtjz\" (UID: \"ebee721b-f5ea-448d-adf8-5cf2398fa566\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-4jtjz"
Jul 27 18:40:08.276097 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:08.276078    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebee721b-f5ea-448d-adf8-5cf2398fa566-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-4jtjz\" (UID: \"ebee721b-f5ea-448d-adf8-5cf2398fa566\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-4jtjz"
Jul 27 18:40:08.285104 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:08.285077    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvkrq\" (UniqueName: \"kubernetes.io/projected/ebee721b-f5ea-448d-adf8-5cf2398fa566-kube-api-access-wvkrq\") pod \"payload-pre-processing-687dbb6cf6-4jtjz\" (UID: \"ebee721b-f5ea-448d-adf8-5cf2398fa566\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-4jtjz"
Jul 27 18:40:08.347780 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:08.347751    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-4jtjz"
Jul 27 18:40:08.359088 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:08.359060    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-557776f786-xvfwc" event={"ID":"af0c2f08-b3af-4f51-944b-22262f3f7e18","Type":"ContainerStarted","Data":"ecf50efb4f6ede20aa5d04719b68e14f7c05c28045b6caca61e990c50ab084d0"}
Jul 27 18:40:08.500143 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:08.497350    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-4jtjz"]
Jul 27 18:40:08.507869 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:40:08.507840    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebee721b_f5ea_448d_adf8_5cf2398fa566.slice/crio-083f140e48fa495396813cca9a9db95b6f8729e388bb87fccdeca3b74e49825f WatchSource:0}: Error finding container 083f140e48fa495396813cca9a9db95b6f8729e388bb87fccdeca3b74e49825f: Status 404 returned error can't find the container with id 083f140e48fa495396813cca9a9db95b6f8729e388bb87fccdeca3b74e49825f
Jul 27 18:40:09.372908 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:09.372864    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-4jtjz" event={"ID":"ebee721b-f5ea-448d-adf8-5cf2398fa566","Type":"ContainerStarted","Data":"083f140e48fa495396813cca9a9db95b6f8729e388bb87fccdeca3b74e49825f"}
Jul 27 18:40:10.381953 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:10.381912    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-557776f786-xvfwc" event={"ID":"af0c2f08-b3af-4f51-944b-22262f3f7e18","Type":"ContainerStarted","Data":"d4f01b06f43a2d63a4cc7ce2da01a410eaf4033539e50bb1d47ec966f710e902"}
Jul 27 18:40:10.382406 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:10.381983    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-557776f786-xvfwc"
Jul 27 18:40:10.404797 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:10.404750    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-557776f786-xvfwc" podStartSLOduration=1.26537084 podStartE2EDuration="3.404734432s" totalImagesPullingTime="2.139363592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:40:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:40:10.402843519 +0000 UTC m=+670.837101130" watchObservedRunningTime="2026-07-27 18:40:10.404734432 +0000 UTC m=+670.838992043"
Jul 27 18:40:11.388308 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:11.388274    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-4jtjz" event={"ID":"ebee721b-f5ea-448d-adf8-5cf2398fa566","Type":"ContainerStarted","Data":"b121de3ad3eec19bce2055cd3d26fb23e6a0364dc44e7926b8cccc9282749125"}
Jul 27 18:40:11.388716 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:11.388513    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-4jtjz"
Jul 27 18:40:11.413061 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:11.413010    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-4jtjz" podStartSLOduration=2.448240463 podStartE2EDuration="4.41299675s" totalImagesPullingTime="1.964756287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:40:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:40:11.410589741 +0000 UTC m=+671.844847352" watchObservedRunningTime="2026-07-27 18:40:11.41299675 +0000 UTC m=+671.847254360"
Jul 27 18:40:16.393983 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:16.393957    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-557776f786-xvfwc"
Jul 27 18:40:22.395609 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:40:22.395579    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-4jtjz"
Jul 27 18:41:06.500262 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:06.500225    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf"]
Jul 27 18:41:06.503540 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:06.503519    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf"
Jul 27 18:41:06.507741 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:06.507705    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 27 18:41:06.507879 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:06.507751    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6w4tt\""
Jul 27 18:41:06.508031 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:06.508008    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 27 18:41:06.508178 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:06.508075    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 27 18:41:06.514887 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:06.514864    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf"]
Jul 27 18:41:06.675344 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:06.675309    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aef97511-8f66-4ab2-9b17-a0b0b10b0ff6-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46ggf\" (UID: \"aef97511-8f66-4ab2-9b17-a0b0b10b0ff6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf"
Jul 27 18:41:06.675503 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:06.675363    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aef97511-8f66-4ab2-9b17-a0b0b10b0ff6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46ggf\" (UID: \"aef97511-8f66-4ab2-9b17-a0b0b10b0ff6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf"
Jul 27 18:41:06.675503 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:06.675388    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26rhh\" (UniqueName: \"kubernetes.io/projected/aef97511-8f66-4ab2-9b17-a0b0b10b0ff6-kube-api-access-26rhh\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46ggf\" (UID: \"aef97511-8f66-4ab2-9b17-a0b0b10b0ff6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf"
Jul 27 18:41:06.675503 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:06.675438    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aef97511-8f66-4ab2-9b17-a0b0b10b0ff6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46ggf\" (UID: \"aef97511-8f66-4ab2-9b17-a0b0b10b0ff6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf"
Jul 27 18:41:06.675503 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:06.675484    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aef97511-8f66-4ab2-9b17-a0b0b10b0ff6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46ggf\" (UID: \"aef97511-8f66-4ab2-9b17-a0b0b10b0ff6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf"
Jul 27 18:41:06.675645 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:06.675518    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aef97511-8f66-4ab2-9b17-a0b0b10b0ff6-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46ggf\" (UID: \"aef97511-8f66-4ab2-9b17-a0b0b10b0ff6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf"
Jul 27 18:41:06.775951 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:06.775922    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aef97511-8f66-4ab2-9b17-a0b0b10b0ff6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46ggf\" (UID: \"aef97511-8f66-4ab2-9b17-a0b0b10b0ff6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf"
Jul 27 18:41:06.776101 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:06.775960    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26rhh\" (UniqueName: \"kubernetes.io/projected/aef97511-8f66-4ab2-9b17-a0b0b10b0ff6-kube-api-access-26rhh\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46ggf\" (UID: \"aef97511-8f66-4ab2-9b17-a0b0b10b0ff6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf"
Jul 27 18:41:06.776101 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:06.775982    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aef97511-8f66-4ab2-9b17-a0b0b10b0ff6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46ggf\" (UID: \"aef97511-8f66-4ab2-9b17-a0b0b10b0ff6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf"
Jul 27 18:41:06.776101 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:06.776018    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aef97511-8f66-4ab2-9b17-a0b0b10b0ff6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46ggf\" (UID: \"aef97511-8f66-4ab2-9b17-a0b0b10b0ff6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf"
Jul 27 18:41:06.776101 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:06.776047    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aef97511-8f66-4ab2-9b17-a0b0b10b0ff6-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46ggf\" (UID: \"aef97511-8f66-4ab2-9b17-a0b0b10b0ff6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf"
Jul 27 18:41:06.776101 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:06.776062    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aef97511-8f66-4ab2-9b17-a0b0b10b0ff6-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46ggf\" (UID: \"aef97511-8f66-4ab2-9b17-a0b0b10b0ff6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf"
Jul 27 18:41:06.776435 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:06.776413    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aef97511-8f66-4ab2-9b17-a0b0b10b0ff6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46ggf\" (UID: \"aef97511-8f66-4ab2-9b17-a0b0b10b0ff6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf"
Jul 27 18:41:06.776435 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:06.776430    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aef97511-8f66-4ab2-9b17-a0b0b10b0ff6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46ggf\" (UID: \"aef97511-8f66-4ab2-9b17-a0b0b10b0ff6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf"
Jul 27 18:41:06.776547 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:06.776506    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aef97511-8f66-4ab2-9b17-a0b0b10b0ff6-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46ggf\" (UID: \"aef97511-8f66-4ab2-9b17-a0b0b10b0ff6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf"
Jul 27 18:41:06.778445 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:06.778429    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aef97511-8f66-4ab2-9b17-a0b0b10b0ff6-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46ggf\" (UID: \"aef97511-8f66-4ab2-9b17-a0b0b10b0ff6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf"
Jul 27 18:41:06.778622 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:06.778606    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aef97511-8f66-4ab2-9b17-a0b0b10b0ff6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46ggf\" (UID: \"aef97511-8f66-4ab2-9b17-a0b0b10b0ff6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf"
Jul 27 18:41:06.784483 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:06.784463    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26rhh\" (UniqueName: \"kubernetes.io/projected/aef97511-8f66-4ab2-9b17-a0b0b10b0ff6-kube-api-access-26rhh\") pod \"e2e-trlp-test-simulated-kserve-598466548d-46ggf\" (UID: \"aef97511-8f66-4ab2-9b17-a0b0b10b0ff6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf"
Jul 27 18:41:06.815003 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:06.814976    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf"
Jul 27 18:41:06.951682 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:06.951652    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf"]
Jul 27 18:41:06.953208 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:41:06.953171    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef97511_8f66_4ab2_9b17_a0b0b10b0ff6.slice/crio-af4db9bde14a5dd535a369dd3ae50667f7f7f546ff901a228108ad7ca3d0bf3d WatchSource:0}: Error finding container af4db9bde14a5dd535a369dd3ae50667f7f7f546ff901a228108ad7ca3d0bf3d: Status 404 returned error can't find the container with id af4db9bde14a5dd535a369dd3ae50667f7f7f546ff901a228108ad7ca3d0bf3d
Jul 27 18:41:07.692416 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:07.692373    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf" event={"ID":"aef97511-8f66-4ab2-9b17-a0b0b10b0ff6","Type":"ContainerStarted","Data":"af4db9bde14a5dd535a369dd3ae50667f7f7f546ff901a228108ad7ca3d0bf3d"}
Jul 27 18:41:08.699337 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:08.699302    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf" event={"ID":"aef97511-8f66-4ab2-9b17-a0b0b10b0ff6","Type":"ContainerStarted","Data":"bd26de4d50c1da3b9d0620a353cd1d14426e237507a79edd3d25d27c466f6e3d"}
Jul 27 18:41:08.720605 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:08.720555    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf" podStartSLOduration=1.729695424 podStartE2EDuration="2.720541612s" totalImagesPullingTime="990.846188ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:41:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:41:08.719291468 +0000 UTC m=+729.153549079" watchObservedRunningTime="2026-07-27 18:41:08.720541612 +0000 UTC m=+729.154799222"
Jul 27 18:41:16.815292 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:16.815257    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf"
Jul 27 18:41:16.815821 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:16.815340    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf"
Jul 27 18:41:16.827878 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:16.827857    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf"
Jul 27 18:41:17.757994 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:17.757961    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-46ggf"
Jul 27 18:41:49.793866 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:49.793828    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw"]
Jul 27 18:41:49.798604 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:49.798581    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw"
Jul 27 18:41:49.801722 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:49.801703    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 27 18:41:49.808212 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:49.808189    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw"]
Jul 27 18:41:49.859433 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:49.859398    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27781d28-5639-4a2e-9cd9-e3f5804fa0ca-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw\" (UID: \"27781d28-5639-4a2e-9cd9-e3f5804fa0ca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw"
Jul 27 18:41:49.859544 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:49.859471    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtp4r\" (UniqueName: \"kubernetes.io/projected/27781d28-5639-4a2e-9cd9-e3f5804fa0ca-kube-api-access-vtp4r\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw\" (UID: \"27781d28-5639-4a2e-9cd9-e3f5804fa0ca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw"
Jul 27 18:41:49.859544 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:49.859516    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27781d28-5639-4a2e-9cd9-e3f5804fa0ca-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw\" (UID: \"27781d28-5639-4a2e-9cd9-e3f5804fa0ca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw"
Jul 27 18:41:49.859544 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:49.859532    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27781d28-5639-4a2e-9cd9-e3f5804fa0ca-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw\" (UID: \"27781d28-5639-4a2e-9cd9-e3f5804fa0ca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw"
Jul 27 18:41:49.859666 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:49.859566    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27781d28-5639-4a2e-9cd9-e3f5804fa0ca-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw\" (UID: \"27781d28-5639-4a2e-9cd9-e3f5804fa0ca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw"
Jul 27 18:41:49.859666 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:49.859591    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27781d28-5639-4a2e-9cd9-e3f5804fa0ca-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw\" (UID: \"27781d28-5639-4a2e-9cd9-e3f5804fa0ca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw"
Jul 27 18:41:49.960653 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:49.960619    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27781d28-5639-4a2e-9cd9-e3f5804fa0ca-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw\" (UID: \"27781d28-5639-4a2e-9cd9-e3f5804fa0ca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw"
Jul 27 18:41:49.960653 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:49.960654    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27781d28-5639-4a2e-9cd9-e3f5804fa0ca-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw\" (UID: \"27781d28-5639-4a2e-9cd9-e3f5804fa0ca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw"
Jul 27 18:41:49.960852 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:49.960675    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27781d28-5639-4a2e-9cd9-e3f5804fa0ca-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw\" (UID: \"27781d28-5639-4a2e-9cd9-e3f5804fa0ca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw"
Jul 27 18:41:49.960852 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:49.960697    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27781d28-5639-4a2e-9cd9-e3f5804fa0ca-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw\" (UID: \"27781d28-5639-4a2e-9cd9-e3f5804fa0ca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw"
Jul 27 18:41:49.960943 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:49.960874    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27781d28-5639-4a2e-9cd9-e3f5804fa0ca-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw\" (UID: \"27781d28-5639-4a2e-9cd9-e3f5804fa0ca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw"
Jul 27 18:41:49.961044 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:49.961020    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtp4r\" (UniqueName: \"kubernetes.io/projected/27781d28-5639-4a2e-9cd9-e3f5804fa0ca-kube-api-access-vtp4r\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw\" (UID: \"27781d28-5639-4a2e-9cd9-e3f5804fa0ca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw"
Jul 27 18:41:49.961153 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:49.961044    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27781d28-5639-4a2e-9cd9-e3f5804fa0ca-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw\" (UID: \"27781d28-5639-4a2e-9cd9-e3f5804fa0ca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw"
Jul 27 18:41:49.961153 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:49.961141    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27781d28-5639-4a2e-9cd9-e3f5804fa0ca-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw\" (UID: \"27781d28-5639-4a2e-9cd9-e3f5804fa0ca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw"
Jul 27 18:41:49.961330 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:49.961284    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27781d28-5639-4a2e-9cd9-e3f5804fa0ca-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw\" (UID: \"27781d28-5639-4a2e-9cd9-e3f5804fa0ca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw"
Jul 27 18:41:49.963187 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:49.963163    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27781d28-5639-4a2e-9cd9-e3f5804fa0ca-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw\" (UID: \"27781d28-5639-4a2e-9cd9-e3f5804fa0ca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw"
Jul 27 18:41:49.963339 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:49.963323    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27781d28-5639-4a2e-9cd9-e3f5804fa0ca-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw\" (UID: \"27781d28-5639-4a2e-9cd9-e3f5804fa0ca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw"
Jul 27 18:41:49.972169 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:49.972148    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtp4r\" (UniqueName: \"kubernetes.io/projected/27781d28-5639-4a2e-9cd9-e3f5804fa0ca-kube-api-access-vtp4r\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw\" (UID: \"27781d28-5639-4a2e-9cd9-e3f5804fa0ca\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw"
Jul 27 18:41:50.108432 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:50.108347    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw"
Jul 27 18:41:50.256753 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:50.256730    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw"]
Jul 27 18:41:50.258572 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:41:50.258542    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27781d28_5639_4a2e_9cd9_e3f5804fa0ca.slice/crio-fa2ea9f6930b8046b468217c6cec857e57da0e43f814db3029665f343a5b7349 WatchSource:0}: Error finding container fa2ea9f6930b8046b468217c6cec857e57da0e43f814db3029665f343a5b7349: Status 404 returned error can't find the container with id fa2ea9f6930b8046b468217c6cec857e57da0e43f814db3029665f343a5b7349
Jul 27 18:41:50.920392 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:50.920357    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw" event={"ID":"27781d28-5639-4a2e-9cd9-e3f5804fa0ca","Type":"ContainerStarted","Data":"7723fbb70973ec8e590bb286f4e62f7477271358751404fc3b05c6baa1f4ee75"}
Jul 27 18:41:50.920784 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:50.920399    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw" event={"ID":"27781d28-5639-4a2e-9cd9-e3f5804fa0ca","Type":"ContainerStarted","Data":"fa2ea9f6930b8046b468217c6cec857e57da0e43f814db3029665f343a5b7349"}
Jul 27 18:41:50.941798 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:41:50.941748    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw" podStartSLOduration=1.749789166 podStartE2EDuration="1.941734188s" totalImagesPullingTime="191.945022ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:41:50.940240058 +0000 UTC m=+771.374497669" watchObservedRunningTime="2026-07-27 18:41:50.941734188 +0000 UTC m=+771.375991799"
Jul 27 18:42:00.108981 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:42:00.108939    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw"
Jul 27 18:42:00.109421 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:42:00.108997    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw"
Jul 27 18:42:00.125335 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:42:00.125310    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw"
Jul 27 18:42:00.985969 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:42:00.985941    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw"
Jul 27 18:42:49.390425 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:42:49.390346    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7dc5bb56d-js2x6"]
Jul 27 18:42:49.393856 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:42:49.393839    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dc5bb56d-js2x6"
Jul 27 18:42:49.396745 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:42:49.396718    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-d825p\""
Jul 27 18:42:49.398729 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:42:49.398713    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 27 18:42:49.404956 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:42:49.404930    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7dc5bb56d-js2x6"]
Jul 27 18:42:49.498851 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:42:49.498817    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-862hr\" (UniqueName: \"kubernetes.io/projected/facc30c3-4d3f-4939-8568-4581518e23a2-kube-api-access-862hr\") pod \"authorino-7dc5bb56d-js2x6\" (UID: \"facc30c3-4d3f-4939-8568-4581518e23a2\") " pod="kuadrant-system/authorino-7dc5bb56d-js2x6"
Jul 27 18:42:49.499004 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:42:49.498910    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/facc30c3-4d3f-4939-8568-4581518e23a2-tls-cert\") pod \"authorino-7dc5bb56d-js2x6\" (UID: \"facc30c3-4d3f-4939-8568-4581518e23a2\") " pod="kuadrant-system/authorino-7dc5bb56d-js2x6"
Jul 27 18:42:49.599361 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:42:49.599323    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/facc30c3-4d3f-4939-8568-4581518e23a2-tls-cert\") pod \"authorino-7dc5bb56d-js2x6\" (UID: \"facc30c3-4d3f-4939-8568-4581518e23a2\") " pod="kuadrant-system/authorino-7dc5bb56d-js2x6"
Jul 27 18:42:49.599515 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:42:49.599387    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-862hr\" (UniqueName: \"kubernetes.io/projected/facc30c3-4d3f-4939-8568-4581518e23a2-kube-api-access-862hr\") pod \"authorino-7dc5bb56d-js2x6\" (UID: \"facc30c3-4d3f-4939-8568-4581518e23a2\") " pod="kuadrant-system/authorino-7dc5bb56d-js2x6"
Jul 27 18:42:49.601969 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:42:49.601942    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/facc30c3-4d3f-4939-8568-4581518e23a2-tls-cert\") pod \"authorino-7dc5bb56d-js2x6\" (UID: \"facc30c3-4d3f-4939-8568-4581518e23a2\") " pod="kuadrant-system/authorino-7dc5bb56d-js2x6"
Jul 27 18:42:49.609280 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:42:49.609251    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-862hr\" (UniqueName: \"kubernetes.io/projected/facc30c3-4d3f-4939-8568-4581518e23a2-kube-api-access-862hr\") pod \"authorino-7dc5bb56d-js2x6\" (UID: \"facc30c3-4d3f-4939-8568-4581518e23a2\") " pod="kuadrant-system/authorino-7dc5bb56d-js2x6"
Jul 27 18:42:49.702989 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:42:49.702931    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dc5bb56d-js2x6"
Jul 27 18:42:49.850287 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:42:49.850261    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7dc5bb56d-js2x6"]
Jul 27 18:42:49.851732 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:42:49.851704    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfacc30c3_4d3f_4939_8568_4581518e23a2.slice/crio-3b7cb5abbe24ec1566627e12a3c036f482831472a072425e518de4a12cbc6910 WatchSource:0}: Error finding container 3b7cb5abbe24ec1566627e12a3c036f482831472a072425e518de4a12cbc6910: Status 404 returned error can't find the container with id 3b7cb5abbe24ec1566627e12a3c036f482831472a072425e518de4a12cbc6910
Jul 27 18:42:50.239686 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:42:50.239655    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dc5bb56d-js2x6" event={"ID":"facc30c3-4d3f-4939-8568-4581518e23a2","Type":"ContainerStarted","Data":"3b7cb5abbe24ec1566627e12a3c036f482831472a072425e518de4a12cbc6910"}
Jul 27 18:42:51.247575 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:42:51.247534    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dc5bb56d-js2x6" event={"ID":"facc30c3-4d3f-4939-8568-4581518e23a2","Type":"ContainerStarted","Data":"9ae1e9fee35f919298b7610f10f146201777913227c6076b06a44c00dd233506"}
Jul 27 18:42:51.281967 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:42:51.281912    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7dc5bb56d-js2x6" podStartSLOduration=1.695730083 podStartE2EDuration="2.281895854s" totalImagesPullingTime="586.165771ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:42:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:42:51.281638782 +0000 UTC m=+831.715896395" watchObservedRunningTime="2026-07-27 18:42:51.281895854 +0000 UTC m=+831.716153465"
Jul 27 18:43:37.666419 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:43:37.666377    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77ff6dccdd-424r7"]
Jul 27 18:43:37.666963 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:43:37.666599    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-77ff6dccdd-424r7" podUID="47855e8a-9217-45dc-83fa-830c9928128e" containerName="manager" containerID="cri-o://de2c7608e3a51bcdb0d377b72ef10058743e9e90933c58841ed7a7ff303ebc35" gracePeriod=30
Jul 27 18:43:37.918649 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:43:37.918589    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77ff6dccdd-424r7"
Jul 27 18:43:38.050622 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:43:38.050593    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lk92t\" (UniqueName: \"kubernetes.io/projected/47855e8a-9217-45dc-83fa-830c9928128e-kube-api-access-lk92t\") pod \"47855e8a-9217-45dc-83fa-830c9928128e\" (UID: \"47855e8a-9217-45dc-83fa-830c9928128e\") "
Jul 27 18:43:38.050791 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:43:38.050652    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47855e8a-9217-45dc-83fa-830c9928128e-webhook-cert\") pod \"47855e8a-9217-45dc-83fa-830c9928128e\" (UID: \"47855e8a-9217-45dc-83fa-830c9928128e\") "
Jul 27 18:43:38.052980 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:43:38.052942    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47855e8a-9217-45dc-83fa-830c9928128e-kube-api-access-lk92t" (OuterVolumeSpecName: "kube-api-access-lk92t") pod "47855e8a-9217-45dc-83fa-830c9928128e" (UID: "47855e8a-9217-45dc-83fa-830c9928128e"). InnerVolumeSpecName "kube-api-access-lk92t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:43:38.053103 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:43:38.052980    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47855e8a-9217-45dc-83fa-830c9928128e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "47855e8a-9217-45dc-83fa-830c9928128e" (UID: "47855e8a-9217-45dc-83fa-830c9928128e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:43:38.152270 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:43:38.152240    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk92t\" (UniqueName: \"kubernetes.io/projected/47855e8a-9217-45dc-83fa-830c9928128e-kube-api-access-lk92t\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:43:38.152270 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:43:38.152264    2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47855e8a-9217-45dc-83fa-830c9928128e-webhook-cert\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:43:38.506561 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:43:38.506524    2566 generic.go:356] "Generic (PLEG): container finished" podID="47855e8a-9217-45dc-83fa-830c9928128e" containerID="de2c7608e3a51bcdb0d377b72ef10058743e9e90933c58841ed7a7ff303ebc35" exitCode=0
Jul 27 18:43:38.506731 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:43:38.506567    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77ff6dccdd-424r7"
Jul 27 18:43:38.506731 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:43:38.506576    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77ff6dccdd-424r7" event={"ID":"47855e8a-9217-45dc-83fa-830c9928128e","Type":"ContainerDied","Data":"de2c7608e3a51bcdb0d377b72ef10058743e9e90933c58841ed7a7ff303ebc35"}
Jul 27 18:43:38.506731 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:43:38.506621    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77ff6dccdd-424r7" event={"ID":"47855e8a-9217-45dc-83fa-830c9928128e","Type":"ContainerDied","Data":"de8b26fbe1fe6fcb68cafd1a29346530c01c610841b2872e8890b4297e756363"}
Jul 27 18:43:38.506731 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:43:38.506645    2566 scope.go:117] "RemoveContainer" containerID="de2c7608e3a51bcdb0d377b72ef10058743e9e90933c58841ed7a7ff303ebc35"
Jul 27 18:43:38.515618 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:43:38.515601    2566 scope.go:117] "RemoveContainer" containerID="de2c7608e3a51bcdb0d377b72ef10058743e9e90933c58841ed7a7ff303ebc35"
Jul 27 18:43:38.515887 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:43:38.515870    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de2c7608e3a51bcdb0d377b72ef10058743e9e90933c58841ed7a7ff303ebc35\": container with ID starting with de2c7608e3a51bcdb0d377b72ef10058743e9e90933c58841ed7a7ff303ebc35 not found: ID does not exist" containerID="de2c7608e3a51bcdb0d377b72ef10058743e9e90933c58841ed7a7ff303ebc35"
Jul 27 18:43:38.515940 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:43:38.515895    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de2c7608e3a51bcdb0d377b72ef10058743e9e90933c58841ed7a7ff303ebc35"} err="failed to get container status \"de2c7608e3a51bcdb0d377b72ef10058743e9e90933c58841ed7a7ff303ebc35\": rpc error: code = NotFound desc = could not find container \"de2c7608e3a51bcdb0d377b72ef10058743e9e90933c58841ed7a7ff303ebc35\": container with ID starting with de2c7608e3a51bcdb0d377b72ef10058743e9e90933c58841ed7a7ff303ebc35 not found: ID does not exist"
Jul 27 18:43:38.527676 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:43:38.527653    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77ff6dccdd-424r7"]
Jul 27 18:43:38.535505 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:43:38.535484    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-77ff6dccdd-424r7"]
Jul 27 18:43:40.123692 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:43:40.123662    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47855e8a-9217-45dc-83fa-830c9928128e" path="/var/lib/kubelet/pods/47855e8a-9217-45dc-83fa-830c9928128e/volumes"
Jul 27 18:44:00.045085 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:44:00.045058    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 18:44:00.047539 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:44:00.047521    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 18:49:00.087844 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:49:00.087805    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 18:49:00.088741 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:49:00.088715    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 18:53:36.211129 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.211080    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"]
Jul 27 18:53:36.211541 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.211497    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47855e8a-9217-45dc-83fa-830c9928128e" containerName="manager"
Jul 27 18:53:36.211541 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.211510    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="47855e8a-9217-45dc-83fa-830c9928128e" containerName="manager"
Jul 27 18:53:36.211615 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.211588    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="47855e8a-9217-45dc-83fa-830c9928128e" containerName="manager"
Jul 27 18:53:36.214593 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.214578    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"
Jul 27 18:53:36.217792 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.217770    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-a92e6c29-kserve-self-signed-certs\""
Jul 27 18:53:36.248641 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.248608    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/980bc239-79b7-4b31-889e-86f9e0785c30-dshm\") pod \"e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk\" (UID: \"980bc239-79b7-4b31-889e-86f9e0785c30\") " pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"
Jul 27 18:53:36.248641 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.248643    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/980bc239-79b7-4b31-889e-86f9e0785c30-model-cache\") pod \"e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk\" (UID: \"980bc239-79b7-4b31-889e-86f9e0785c30\") " pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"
Jul 27 18:53:36.248832 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.248811    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/980bc239-79b7-4b31-889e-86f9e0785c30-tls-certs\") pod \"e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk\" (UID: \"980bc239-79b7-4b31-889e-86f9e0785c30\") " pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"
Jul 27 18:53:36.248918 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.248895    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/980bc239-79b7-4b31-889e-86f9e0785c30-tmp-dir\") pod \"e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk\" (UID: \"980bc239-79b7-4b31-889e-86f9e0785c30\") " pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"
Jul 27 18:53:36.249002 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.248973    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/980bc239-79b7-4b31-889e-86f9e0785c30-home\") pod \"e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk\" (UID: \"980bc239-79b7-4b31-889e-86f9e0785c30\") " pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"
Jul 27 18:53:36.249067 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.249008    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqsgl\" (UniqueName: \"kubernetes.io/projected/980bc239-79b7-4b31-889e-86f9e0785c30-kube-api-access-hqsgl\") pod \"e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk\" (UID: \"980bc239-79b7-4b31-889e-86f9e0785c30\") " pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"
Jul 27 18:53:36.252203 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.252169    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"]
Jul 27 18:53:36.349866 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.349834    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/980bc239-79b7-4b31-889e-86f9e0785c30-tmp-dir\") pod \"e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk\" (UID: \"980bc239-79b7-4b31-889e-86f9e0785c30\") " pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"
Jul 27 18:53:36.350053 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.349878    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/980bc239-79b7-4b31-889e-86f9e0785c30-home\") pod \"e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk\" (UID: \"980bc239-79b7-4b31-889e-86f9e0785c30\") " pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"
Jul 27 18:53:36.350053 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.349895    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqsgl\" (UniqueName: \"kubernetes.io/projected/980bc239-79b7-4b31-889e-86f9e0785c30-kube-api-access-hqsgl\") pod \"e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk\" (UID: \"980bc239-79b7-4b31-889e-86f9e0785c30\") " pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"
Jul 27 18:53:36.350053 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.349916    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/980bc239-79b7-4b31-889e-86f9e0785c30-dshm\") pod \"e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk\" (UID: \"980bc239-79b7-4b31-889e-86f9e0785c30\") " pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"
Jul 27 18:53:36.350053 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.349937    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/980bc239-79b7-4b31-889e-86f9e0785c30-model-cache\") pod \"e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk\" (UID: \"980bc239-79b7-4b31-889e-86f9e0785c30\") " pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"
Jul 27 18:53:36.350053 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.350029    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/980bc239-79b7-4b31-889e-86f9e0785c30-tls-certs\") pod \"e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk\" (UID: \"980bc239-79b7-4b31-889e-86f9e0785c30\") " pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"
Jul 27 18:53:36.350356 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.350278    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/980bc239-79b7-4b31-889e-86f9e0785c30-tmp-dir\") pod \"e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk\" (UID: \"980bc239-79b7-4b31-889e-86f9e0785c30\") " pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"
Jul 27 18:53:36.350414 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.350384    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/980bc239-79b7-4b31-889e-86f9e0785c30-home\") pod \"e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk\" (UID: \"980bc239-79b7-4b31-889e-86f9e0785c30\") " pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"
Jul 27 18:53:36.350458 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.350412    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/980bc239-79b7-4b31-889e-86f9e0785c30-model-cache\") pod \"e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk\" (UID: \"980bc239-79b7-4b31-889e-86f9e0785c30\") " pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"
Jul 27 18:53:36.352181 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.352157    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/980bc239-79b7-4b31-889e-86f9e0785c30-dshm\") pod \"e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk\" (UID: \"980bc239-79b7-4b31-889e-86f9e0785c30\") " pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"
Jul 27 18:53:36.352524 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.352508    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/980bc239-79b7-4b31-889e-86f9e0785c30-tls-certs\") pod \"e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk\" (UID: \"980bc239-79b7-4b31-889e-86f9e0785c30\") " pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"
Jul 27 18:53:36.359736 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.359713    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqsgl\" (UniqueName: \"kubernetes.io/projected/980bc239-79b7-4b31-889e-86f9e0785c30-kube-api-access-hqsgl\") pod \"e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk\" (UID: \"980bc239-79b7-4b31-889e-86f9e0785c30\") " pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"
Jul 27 18:53:36.523881 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.523848    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"
Jul 27 18:53:36.668522 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.668492    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"]
Jul 27 18:53:36.669550 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:53:36.669523    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod980bc239_79b7_4b31_889e_86f9e0785c30.slice/crio-4cfc92edc0d42c586aa2c4f40f2d66386ab610c78494da506843988a7981adc4 WatchSource:0}: Error finding container 4cfc92edc0d42c586aa2c4f40f2d66386ab610c78494da506843988a7981adc4: Status 404 returned error can't find the container with id 4cfc92edc0d42c586aa2c4f40f2d66386ab610c78494da506843988a7981adc4
Jul 27 18:53:36.721130 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:36.721084    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk" event={"ID":"980bc239-79b7-4b31-889e-86f9e0785c30","Type":"ContainerStarted","Data":"4cfc92edc0d42c586aa2c4f40f2d66386ab610c78494da506843988a7981adc4"}
Jul 27 18:53:37.727933 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:37.727898    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk" event={"ID":"980bc239-79b7-4b31-889e-86f9e0785c30","Type":"ContainerStarted","Data":"55dfb28e10eec9f2c4ddae6b111092d7ad69abf5b55d9792712d9d8dcbdac138"}
Jul 27 18:53:37.752080 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:37.752011    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk" podStartSLOduration=1.751990452 podStartE2EDuration="1.751990452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:53:37.748557647 +0000 UTC m=+1478.182815259" watchObservedRunningTime="2026-07-27 18:53:37.751990452 +0000 UTC m=+1478.186248068"
Jul 27 18:53:46.524532 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:46.524498    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"
Jul 27 18:53:46.524946 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:46.524548    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"
Jul 27 18:53:46.542859 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:46.542827    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"
Jul 27 18:53:46.786656 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:53:46.786575    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"
Jul 27 18:54:00.130253 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:00.130226    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 18:54:00.141431 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:00.141408    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 18:54:31.157994 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:31.157918    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"]
Jul 27 18:54:46.177262 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:46.177218    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk" podUID="980bc239-79b7-4b31-889e-86f9e0785c30" containerName="main" containerID="cri-o://55dfb28e10eec9f2c4ddae6b111092d7ad69abf5b55d9792712d9d8dcbdac138" gracePeriod=45
Jul 27 18:54:46.426290 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:46.426266    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"
Jul 27 18:54:46.599236 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:46.599202    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/980bc239-79b7-4b31-889e-86f9e0785c30-dshm\") pod \"980bc239-79b7-4b31-889e-86f9e0785c30\" (UID: \"980bc239-79b7-4b31-889e-86f9e0785c30\") "
Jul 27 18:54:46.599426 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:46.599258    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/980bc239-79b7-4b31-889e-86f9e0785c30-tls-certs\") pod \"980bc239-79b7-4b31-889e-86f9e0785c30\" (UID: \"980bc239-79b7-4b31-889e-86f9e0785c30\") "
Jul 27 18:54:46.599426 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:46.599296    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/980bc239-79b7-4b31-889e-86f9e0785c30-home\") pod \"980bc239-79b7-4b31-889e-86f9e0785c30\" (UID: \"980bc239-79b7-4b31-889e-86f9e0785c30\") "
Jul 27 18:54:46.599426 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:46.599340    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/980bc239-79b7-4b31-889e-86f9e0785c30-model-cache\") pod \"980bc239-79b7-4b31-889e-86f9e0785c30\" (UID: \"980bc239-79b7-4b31-889e-86f9e0785c30\") "
Jul 27 18:54:46.599426 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:46.599365    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqsgl\" (UniqueName: \"kubernetes.io/projected/980bc239-79b7-4b31-889e-86f9e0785c30-kube-api-access-hqsgl\") pod \"980bc239-79b7-4b31-889e-86f9e0785c30\" (UID: \"980bc239-79b7-4b31-889e-86f9e0785c30\") "
Jul 27 18:54:46.599636 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:46.599451    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/980bc239-79b7-4b31-889e-86f9e0785c30-tmp-dir\") pod \"980bc239-79b7-4b31-889e-86f9e0785c30\" (UID: \"980bc239-79b7-4b31-889e-86f9e0785c30\") "
Jul 27 18:54:46.599636 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:46.599611    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/980bc239-79b7-4b31-889e-86f9e0785c30-home" (OuterVolumeSpecName: "home") pod "980bc239-79b7-4b31-889e-86f9e0785c30" (UID: "980bc239-79b7-4b31-889e-86f9e0785c30"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:54:46.599752 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:46.599615    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/980bc239-79b7-4b31-889e-86f9e0785c30-model-cache" (OuterVolumeSpecName: "model-cache") pod "980bc239-79b7-4b31-889e-86f9e0785c30" (UID: "980bc239-79b7-4b31-889e-86f9e0785c30"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:54:46.599752 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:46.599733    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/980bc239-79b7-4b31-889e-86f9e0785c30-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "980bc239-79b7-4b31-889e-86f9e0785c30" (UID: "980bc239-79b7-4b31-889e-86f9e0785c30"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:54:46.599861 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:46.599783    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/980bc239-79b7-4b31-889e-86f9e0785c30-home\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:54:46.599861 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:46.599795    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/980bc239-79b7-4b31-889e-86f9e0785c30-model-cache\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:54:46.601692 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:46.601664    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/980bc239-79b7-4b31-889e-86f9e0785c30-kube-api-access-hqsgl" (OuterVolumeSpecName: "kube-api-access-hqsgl") pod "980bc239-79b7-4b31-889e-86f9e0785c30" (UID: "980bc239-79b7-4b31-889e-86f9e0785c30"). InnerVolumeSpecName "kube-api-access-hqsgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:54:46.601805 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:46.601693    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/980bc239-79b7-4b31-889e-86f9e0785c30-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "980bc239-79b7-4b31-889e-86f9e0785c30" (UID: "980bc239-79b7-4b31-889e-86f9e0785c30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:54:46.601805 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:46.601723    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/980bc239-79b7-4b31-889e-86f9e0785c30-dshm" (OuterVolumeSpecName: "dshm") pod "980bc239-79b7-4b31-889e-86f9e0785c30" (UID: "980bc239-79b7-4b31-889e-86f9e0785c30"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:54:46.701219 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:46.701184    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/980bc239-79b7-4b31-889e-86f9e0785c30-tmp-dir\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:54:46.701219 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:46.701212    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/980bc239-79b7-4b31-889e-86f9e0785c30-dshm\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:54:46.701219 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:46.701221    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/980bc239-79b7-4b31-889e-86f9e0785c30-tls-certs\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:54:46.701219 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:46.701230    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqsgl\" (UniqueName: \"kubernetes.io/projected/980bc239-79b7-4b31-889e-86f9e0785c30-kube-api-access-hqsgl\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:54:47.100642 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:47.100608    2566 generic.go:356] "Generic (PLEG): container finished" podID="980bc239-79b7-4b31-889e-86f9e0785c30" containerID="55dfb28e10eec9f2c4ddae6b111092d7ad69abf5b55d9792712d9d8dcbdac138" exitCode=0
Jul 27 18:54:47.100830 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:47.100646    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"
Jul 27 18:54:47.100830 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:47.100698    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk" event={"ID":"980bc239-79b7-4b31-889e-86f9e0785c30","Type":"ContainerDied","Data":"55dfb28e10eec9f2c4ddae6b111092d7ad69abf5b55d9792712d9d8dcbdac138"}
Jul 27 18:54:47.100830 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:47.100733    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk" event={"ID":"980bc239-79b7-4b31-889e-86f9e0785c30","Type":"ContainerDied","Data":"4cfc92edc0d42c586aa2c4f40f2d66386ab610c78494da506843988a7981adc4"}
Jul 27 18:54:47.100830 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:47.100749    2566 scope.go:117] "RemoveContainer" containerID="55dfb28e10eec9f2c4ddae6b111092d7ad69abf5b55d9792712d9d8dcbdac138"
Jul 27 18:54:47.110165 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:47.110147    2566 scope.go:117] "RemoveContainer" containerID="55dfb28e10eec9f2c4ddae6b111092d7ad69abf5b55d9792712d9d8dcbdac138"
Jul 27 18:54:47.110431 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:54:47.110413    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55dfb28e10eec9f2c4ddae6b111092d7ad69abf5b55d9792712d9d8dcbdac138\": container with ID starting with 55dfb28e10eec9f2c4ddae6b111092d7ad69abf5b55d9792712d9d8dcbdac138 not found: ID does not exist" containerID="55dfb28e10eec9f2c4ddae6b111092d7ad69abf5b55d9792712d9d8dcbdac138"
Jul 27 18:54:47.110473 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:47.110441    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55dfb28e10eec9f2c4ddae6b111092d7ad69abf5b55d9792712d9d8dcbdac138"} err="failed to get container status \"55dfb28e10eec9f2c4ddae6b111092d7ad69abf5b55d9792712d9d8dcbdac138\": rpc error: code = NotFound desc = could not find container \"55dfb28e10eec9f2c4ddae6b111092d7ad69abf5b55d9792712d9d8dcbdac138\": container with ID starting with 55dfb28e10eec9f2c4ddae6b111092d7ad69abf5b55d9792712d9d8dcbdac138 not found: ID does not exist"
Jul 27 18:54:47.126981 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:47.126959    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"]
Jul 27 18:54:47.129154 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:47.129135    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-a92e6c29-kserve-696fd448f9-zpmqk"]
Jul 27 18:54:48.123573 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:54:48.123536    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="980bc239-79b7-4b31-889e-86f9e0785c30" path="/var/lib/kubelet/pods/980bc239-79b7-4b31-889e-86f9e0785c30/volumes"
Jul 27 18:58:14.799462 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:14.799431    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5"]
Jul 27 18:58:14.799939 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:14.799804    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="980bc239-79b7-4b31-889e-86f9e0785c30" containerName="main"
Jul 27 18:58:14.799939 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:14.799814    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="980bc239-79b7-4b31-889e-86f9e0785c30" containerName="main"
Jul 27 18:58:14.799939 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:14.799887    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="980bc239-79b7-4b31-889e-86f9e0785c30" containerName="main"
Jul 27 18:58:14.802925 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:14.802909    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5"
Jul 27 18:58:14.806362 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:14.806345    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-001f5ecd-serving-cert\""
Jul 27 18:58:14.806471 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:14.806450    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b9bn6\""
Jul 27 18:58:14.815324 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:14.815300    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5"]
Jul 27 18:58:14.863035 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:14.863006    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2671e44-29dd-4c5c-b935-a4906d9e2946-openshift-service-ca\") pod \"maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5\" (UID: \"a2671e44-29dd-4c5c-b935-a4906d9e2946\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5"
Jul 27 18:58:14.863199 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:14.863053    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjpjj\" (UniqueName: \"kubernetes.io/projected/a2671e44-29dd-4c5c-b935-a4906d9e2946-kube-api-access-wjpjj\") pod \"maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5\" (UID: \"a2671e44-29dd-4c5c-b935-a4906d9e2946\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5"
Jul 27 18:58:14.863199 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:14.863072    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2671e44-29dd-4c5c-b935-a4906d9e2946-maas-api-tls\") pod \"maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5\" (UID: \"a2671e44-29dd-4c5c-b935-a4906d9e2946\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5"
Jul 27 18:58:14.963952 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:14.963915    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2671e44-29dd-4c5c-b935-a4906d9e2946-openshift-service-ca\") pod \"maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5\" (UID: \"a2671e44-29dd-4c5c-b935-a4906d9e2946\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5"
Jul 27 18:58:14.964151 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:14.963975    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjpjj\" (UniqueName: \"kubernetes.io/projected/a2671e44-29dd-4c5c-b935-a4906d9e2946-kube-api-access-wjpjj\") pod \"maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5\" (UID: \"a2671e44-29dd-4c5c-b935-a4906d9e2946\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5"
Jul 27 18:58:14.964151 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:14.964013    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2671e44-29dd-4c5c-b935-a4906d9e2946-maas-api-tls\") pod \"maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5\" (UID: \"a2671e44-29dd-4c5c-b935-a4906d9e2946\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5"
Jul 27 18:58:14.964292 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:58:14.964163    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-serving-cert: secret "maas-api-e2e-ait-001f5ecd-serving-cert" not found
Jul 27 18:58:14.964292 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:58:14.964247    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2671e44-29dd-4c5c-b935-a4906d9e2946-maas-api-tls podName:a2671e44-29dd-4c5c-b935-a4906d9e2946 nodeName:}" failed. No retries permitted until 2026-07-27 18:58:15.464225207 +0000 UTC m=+1755.898482811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a2671e44-29dd-4c5c-b935-a4906d9e2946-maas-api-tls") pod "maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5" (UID: "a2671e44-29dd-4c5c-b935-a4906d9e2946") : secret "maas-api-e2e-ait-001f5ecd-serving-cert" not found
Jul 27 18:58:14.964649 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:14.964632    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2671e44-29dd-4c5c-b935-a4906d9e2946-openshift-service-ca\") pod \"maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5\" (UID: \"a2671e44-29dd-4c5c-b935-a4906d9e2946\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5"
Jul 27 18:58:14.978238 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:14.978208    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjpjj\" (UniqueName: \"kubernetes.io/projected/a2671e44-29dd-4c5c-b935-a4906d9e2946-kube-api-access-wjpjj\") pod \"maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5\" (UID: \"a2671e44-29dd-4c5c-b935-a4906d9e2946\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5"
Jul 27 18:58:15.467410 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:15.467377    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2671e44-29dd-4c5c-b935-a4906d9e2946-maas-api-tls\") pod \"maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5\" (UID: \"a2671e44-29dd-4c5c-b935-a4906d9e2946\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5"
Jul 27 18:58:15.470021 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:15.469999    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2671e44-29dd-4c5c-b935-a4906d9e2946-maas-api-tls\") pod \"maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5\" (UID: \"a2671e44-29dd-4c5c-b935-a4906d9e2946\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5"
Jul 27 18:58:15.713191 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:15.713153    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5"
Jul 27 18:58:15.866145 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:15.866094    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5"]
Jul 27 18:58:15.867941 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:58:15.867917    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2671e44_29dd_4c5c_b935_a4906d9e2946.slice/crio-a15fb980d5fb791fc75cc19a67cf05f5e60a496b17ca5dfa04f74d3dd1db3b91 WatchSource:0}: Error finding container a15fb980d5fb791fc75cc19a67cf05f5e60a496b17ca5dfa04f74d3dd1db3b91: Status 404 returned error can't find the container with id a15fb980d5fb791fc75cc19a67cf05f5e60a496b17ca5dfa04f74d3dd1db3b91
Jul 27 18:58:15.869276 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:15.869257    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 18:58:16.238780 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:16.238743    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5" event={"ID":"a2671e44-29dd-4c5c-b935-a4906d9e2946","Type":"ContainerStarted","Data":"a15fb980d5fb791fc75cc19a67cf05f5e60a496b17ca5dfa04f74d3dd1db3b91"}
Jul 27 18:58:17.245318 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:17.245274    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5" event={"ID":"a2671e44-29dd-4c5c-b935-a4906d9e2946","Type":"ContainerStarted","Data":"552a619877aba8c1b83d2cfb717005fb48401e7c69273b5a6cced6c8ecb4db11"}
Jul 27 18:58:17.245688 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:17.245435    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5"
Jul 27 18:58:17.273835 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:17.273789    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5" podStartSLOduration=2.654467043 podStartE2EDuration="3.273774586s" totalImagesPullingTime="619.307543ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:58:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:58:17.270897822 +0000 UTC m=+1757.705155432" watchObservedRunningTime="2026-07-27 18:58:17.273774586 +0000 UTC m=+1757.708032197"
Jul 27 18:58:23.255986 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:23.255958    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5"
Jul 27 18:58:50.822808 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:50.822775    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5"]
Jul 27 18:58:50.823419 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:50.823066    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5" podUID="a2671e44-29dd-4c5c-b935-a4906d9e2946" containerName="maas-api" containerID="cri-o://552a619877aba8c1b83d2cfb717005fb48401e7c69273b5a6cced6c8ecb4db11" gracePeriod=30
Jul 27 18:58:51.081111 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:51.081053    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5"
Jul 27 18:58:51.206853 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:51.206818    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2671e44-29dd-4c5c-b935-a4906d9e2946-openshift-service-ca\") pod \"a2671e44-29dd-4c5c-b935-a4906d9e2946\" (UID: \"a2671e44-29dd-4c5c-b935-a4906d9e2946\") "
Jul 27 18:58:51.207050 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:51.206904    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjpjj\" (UniqueName: \"kubernetes.io/projected/a2671e44-29dd-4c5c-b935-a4906d9e2946-kube-api-access-wjpjj\") pod \"a2671e44-29dd-4c5c-b935-a4906d9e2946\" (UID: \"a2671e44-29dd-4c5c-b935-a4906d9e2946\") "
Jul 27 18:58:51.207050 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:51.206949    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2671e44-29dd-4c5c-b935-a4906d9e2946-maas-api-tls\") pod \"a2671e44-29dd-4c5c-b935-a4906d9e2946\" (UID: \"a2671e44-29dd-4c5c-b935-a4906d9e2946\") "
Jul 27 18:58:51.207221 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:51.207196    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2671e44-29dd-4c5c-b935-a4906d9e2946-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a2671e44-29dd-4c5c-b935-a4906d9e2946" (UID: "a2671e44-29dd-4c5c-b935-a4906d9e2946"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:58:51.209395 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:51.209367    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2671e44-29dd-4c5c-b935-a4906d9e2946-kube-api-access-wjpjj" (OuterVolumeSpecName: "kube-api-access-wjpjj") pod "a2671e44-29dd-4c5c-b935-a4906d9e2946" (UID: "a2671e44-29dd-4c5c-b935-a4906d9e2946"). InnerVolumeSpecName "kube-api-access-wjpjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:58:51.209482 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:51.209406    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2671e44-29dd-4c5c-b935-a4906d9e2946-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a2671e44-29dd-4c5c-b935-a4906d9e2946" (UID: "a2671e44-29dd-4c5c-b935-a4906d9e2946"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:58:51.307675 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:51.307630    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjpjj\" (UniqueName: \"kubernetes.io/projected/a2671e44-29dd-4c5c-b935-a4906d9e2946-kube-api-access-wjpjj\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:58:51.307675 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:51.307671    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2671e44-29dd-4c5c-b935-a4906d9e2946-maas-api-tls\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:58:51.307886 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:51.307686    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2671e44-29dd-4c5c-b935-a4906d9e2946-openshift-service-ca\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:58:51.442641 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:51.442552    2566 generic.go:356] "Generic (PLEG): container finished" podID="a2671e44-29dd-4c5c-b935-a4906d9e2946" containerID="552a619877aba8c1b83d2cfb717005fb48401e7c69273b5a6cced6c8ecb4db11" exitCode=0
Jul 27 18:58:51.442641 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:51.442602    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5"
Jul 27 18:58:51.442641 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:51.442609    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5" event={"ID":"a2671e44-29dd-4c5c-b935-a4906d9e2946","Type":"ContainerDied","Data":"552a619877aba8c1b83d2cfb717005fb48401e7c69273b5a6cced6c8ecb4db11"}
Jul 27 18:58:51.442853 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:51.442661    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5" event={"ID":"a2671e44-29dd-4c5c-b935-a4906d9e2946","Type":"ContainerDied","Data":"a15fb980d5fb791fc75cc19a67cf05f5e60a496b17ca5dfa04f74d3dd1db3b91"}
Jul 27 18:58:51.442853 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:51.442691    2566 scope.go:117] "RemoveContainer" containerID="552a619877aba8c1b83d2cfb717005fb48401e7c69273b5a6cced6c8ecb4db11"
Jul 27 18:58:51.452468 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:51.452449    2566 scope.go:117] "RemoveContainer" containerID="552a619877aba8c1b83d2cfb717005fb48401e7c69273b5a6cced6c8ecb4db11"
Jul 27 18:58:51.452725 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:58:51.452703    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"552a619877aba8c1b83d2cfb717005fb48401e7c69273b5a6cced6c8ecb4db11\": container with ID starting with 552a619877aba8c1b83d2cfb717005fb48401e7c69273b5a6cced6c8ecb4db11 not found: ID does not exist" containerID="552a619877aba8c1b83d2cfb717005fb48401e7c69273b5a6cced6c8ecb4db11"
Jul 27 18:58:51.452796 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:51.452738    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552a619877aba8c1b83d2cfb717005fb48401e7c69273b5a6cced6c8ecb4db11"} err="failed to get container status \"552a619877aba8c1b83d2cfb717005fb48401e7c69273b5a6cced6c8ecb4db11\": rpc error: code = NotFound desc = could not find container \"552a619877aba8c1b83d2cfb717005fb48401e7c69273b5a6cced6c8ecb4db11\": container with ID starting with 552a619877aba8c1b83d2cfb717005fb48401e7c69273b5a6cced6c8ecb4db11 not found: ID does not exist"
Jul 27 18:58:51.469978 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:51.469944    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5"]
Jul 27 18:58:51.479104 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:51.479077    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-001f5ecd-5db79b64f5-q76g5"]
Jul 27 18:58:52.123703 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:52.123671    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2671e44-29dd-4c5c-b935-a4906d9e2946" path="/var/lib/kubelet/pods/a2671e44-29dd-4c5c-b935-a4906d9e2946/volumes"
Jul 27 18:58:59.092138 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:59.092084    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84"]
Jul 27 18:58:59.092560 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:59.092544    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2671e44-29dd-4c5c-b935-a4906d9e2946" containerName="maas-api"
Jul 27 18:58:59.092606 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:59.092565    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2671e44-29dd-4c5c-b935-a4906d9e2946" containerName="maas-api"
Jul 27 18:58:59.092662 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:59.092652    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2671e44-29dd-4c5c-b935-a4906d9e2946" containerName="maas-api"
Jul 27 18:58:59.095828 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:59.095807    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84"
Jul 27 18:58:59.098997 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:59.098955    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-09eed4cd-serving-cert\""
Jul 27 18:58:59.099159 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:59.099043    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b9bn6\""
Jul 27 18:58:59.108085 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:59.108064    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84"]
Jul 27 18:58:59.175271 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:59.175230    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5jmn\" (UniqueName: \"kubernetes.io/projected/91755248-18ee-49a1-87c0-d6e6141ddce4-kube-api-access-j5jmn\") pod \"maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84\" (UID: \"91755248-18ee-49a1-87c0-d6e6141ddce4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84"
Jul 27 18:58:59.175470 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:59.175294    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91755248-18ee-49a1-87c0-d6e6141ddce4-openshift-service-ca\") pod \"maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84\" (UID: \"91755248-18ee-49a1-87c0-d6e6141ddce4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84"
Jul 27 18:58:59.175470 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:59.175318    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91755248-18ee-49a1-87c0-d6e6141ddce4-maas-api-tls\") pod \"maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84\" (UID: \"91755248-18ee-49a1-87c0-d6e6141ddce4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84"
Jul 27 18:58:59.276717 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:59.276679    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91755248-18ee-49a1-87c0-d6e6141ddce4-openshift-service-ca\") pod \"maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84\" (UID: \"91755248-18ee-49a1-87c0-d6e6141ddce4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84"
Jul 27 18:58:59.276717 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:59.276720    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91755248-18ee-49a1-87c0-d6e6141ddce4-maas-api-tls\") pod \"maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84\" (UID: \"91755248-18ee-49a1-87c0-d6e6141ddce4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84"
Jul 27 18:58:59.276957 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:59.276802    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5jmn\" (UniqueName: \"kubernetes.io/projected/91755248-18ee-49a1-87c0-d6e6141ddce4-kube-api-access-j5jmn\") pod \"maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84\" (UID: \"91755248-18ee-49a1-87c0-d6e6141ddce4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84"
Jul 27 18:58:59.276957 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:58:59.276870    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-serving-cert: secret "maas-api-e2e-migrate-09eed4cd-serving-cert" not found
Jul 27 18:58:59.276957 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:58:59.276954    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91755248-18ee-49a1-87c0-d6e6141ddce4-maas-api-tls podName:91755248-18ee-49a1-87c0-d6e6141ddce4 nodeName:}" failed. No retries permitted until 2026-07-27 18:58:59.776935775 +0000 UTC m=+1800.211193368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/91755248-18ee-49a1-87c0-d6e6141ddce4-maas-api-tls") pod "maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84" (UID: "91755248-18ee-49a1-87c0-d6e6141ddce4") : secret "maas-api-e2e-migrate-09eed4cd-serving-cert" not found
Jul 27 18:58:59.277429 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:59.277409    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91755248-18ee-49a1-87c0-d6e6141ddce4-openshift-service-ca\") pod \"maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84\" (UID: \"91755248-18ee-49a1-87c0-d6e6141ddce4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84"
Jul 27 18:58:59.291944 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:59.291910    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5jmn\" (UniqueName: \"kubernetes.io/projected/91755248-18ee-49a1-87c0-d6e6141ddce4-kube-api-access-j5jmn\") pod \"maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84\" (UID: \"91755248-18ee-49a1-87c0-d6e6141ddce4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84"
Jul 27 18:58:59.782620 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:59.782576    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91755248-18ee-49a1-87c0-d6e6141ddce4-maas-api-tls\") pod \"maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84\" (UID: \"91755248-18ee-49a1-87c0-d6e6141ddce4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84"
Jul 27 18:58:59.785394 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:58:59.785364    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91755248-18ee-49a1-87c0-d6e6141ddce4-maas-api-tls\") pod \"maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84\" (UID: \"91755248-18ee-49a1-87c0-d6e6141ddce4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84"
Jul 27 18:59:00.011257 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:00.011229    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b9bn6\""
Jul 27 18:59:00.018973 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:00.018951    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84"
Jul 27 18:59:00.165513 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:00.165486    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84"]
Jul 27 18:59:00.167850 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:59:00.167820    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91755248_18ee_49a1_87c0_d6e6141ddce4.slice/crio-6c7edabbdcf164314867ed13123c666c22fe9da6772ad5f166742ffa91fd7747 WatchSource:0}: Error finding container 6c7edabbdcf164314867ed13123c666c22fe9da6772ad5f166742ffa91fd7747: Status 404 returned error can't find the container with id 6c7edabbdcf164314867ed13123c666c22fe9da6772ad5f166742ffa91fd7747
Jul 27 18:59:00.176413 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:00.176393    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 18:59:00.185575 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:00.185559    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 18:59:00.492221 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:00.492182    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84" event={"ID":"91755248-18ee-49a1-87c0-d6e6141ddce4","Type":"ContainerStarted","Data":"6c7edabbdcf164314867ed13123c666c22fe9da6772ad5f166742ffa91fd7747"}
Jul 27 18:59:01.499015 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:01.498980    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84" event={"ID":"91755248-18ee-49a1-87c0-d6e6141ddce4","Type":"ContainerStarted","Data":"8af0f2c8066a61b754161be8d4d7d7ac57bceefb8b7f4d16d6f588d7e38585c1"}
Jul 27 18:59:01.499438 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:01.499140    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84"
Jul 27 18:59:01.529429 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:01.529374    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84" podStartSLOduration=2.158694858 podStartE2EDuration="2.529358435s" totalImagesPullingTime="370.663577ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:59:01.527938831 +0000 UTC m=+1801.962196453" watchObservedRunningTime="2026-07-27 18:59:01.529358435 +0000 UTC m=+1801.963616045"
Jul 27 18:59:07.509967 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:07.509931    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84"
Jul 27 18:59:34.614572 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:34.614528    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84"]
Jul 27 18:59:34.615106 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:34.614797    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84" podUID="91755248-18ee-49a1-87c0-d6e6141ddce4" containerName="maas-api" containerID="cri-o://8af0f2c8066a61b754161be8d4d7d7ac57bceefb8b7f4d16d6f588d7e38585c1" gracePeriod=30
Jul 27 18:59:34.868177 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:34.868104    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84"
Jul 27 18:59:35.001192 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:35.001163    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91755248-18ee-49a1-87c0-d6e6141ddce4-openshift-service-ca\") pod \"91755248-18ee-49a1-87c0-d6e6141ddce4\" (UID: \"91755248-18ee-49a1-87c0-d6e6141ddce4\") "
Jul 27 18:59:35.001374 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:35.001210    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91755248-18ee-49a1-87c0-d6e6141ddce4-maas-api-tls\") pod \"91755248-18ee-49a1-87c0-d6e6141ddce4\" (UID: \"91755248-18ee-49a1-87c0-d6e6141ddce4\") "
Jul 27 18:59:35.001374 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:35.001361    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5jmn\" (UniqueName: \"kubernetes.io/projected/91755248-18ee-49a1-87c0-d6e6141ddce4-kube-api-access-j5jmn\") pod \"91755248-18ee-49a1-87c0-d6e6141ddce4\" (UID: \"91755248-18ee-49a1-87c0-d6e6141ddce4\") "
Jul 27 18:59:35.001611 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:35.001588    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91755248-18ee-49a1-87c0-d6e6141ddce4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "91755248-18ee-49a1-87c0-d6e6141ddce4" (UID: "91755248-18ee-49a1-87c0-d6e6141ddce4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:59:35.003553 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:35.003520    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91755248-18ee-49a1-87c0-d6e6141ddce4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "91755248-18ee-49a1-87c0-d6e6141ddce4" (UID: "91755248-18ee-49a1-87c0-d6e6141ddce4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:59:35.003757 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:35.003742    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91755248-18ee-49a1-87c0-d6e6141ddce4-kube-api-access-j5jmn" (OuterVolumeSpecName: "kube-api-access-j5jmn") pod "91755248-18ee-49a1-87c0-d6e6141ddce4" (UID: "91755248-18ee-49a1-87c0-d6e6141ddce4"). InnerVolumeSpecName "kube-api-access-j5jmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:59:35.103000 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:35.102962    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5jmn\" (UniqueName: \"kubernetes.io/projected/91755248-18ee-49a1-87c0-d6e6141ddce4-kube-api-access-j5jmn\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:59:35.103000 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:35.102994    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91755248-18ee-49a1-87c0-d6e6141ddce4-openshift-service-ca\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:59:35.103000 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:35.103004    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91755248-18ee-49a1-87c0-d6e6141ddce4-maas-api-tls\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 18:59:35.686083 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:35.686042    2566 generic.go:356] "Generic (PLEG): container finished" podID="91755248-18ee-49a1-87c0-d6e6141ddce4" containerID="8af0f2c8066a61b754161be8d4d7d7ac57bceefb8b7f4d16d6f588d7e38585c1" exitCode=0
Jul 27 18:59:35.686083 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:35.686081    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84"
Jul 27 18:59:35.686552 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:35.686134    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84" event={"ID":"91755248-18ee-49a1-87c0-d6e6141ddce4","Type":"ContainerDied","Data":"8af0f2c8066a61b754161be8d4d7d7ac57bceefb8b7f4d16d6f588d7e38585c1"}
Jul 27 18:59:35.686552 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:35.686198    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84" event={"ID":"91755248-18ee-49a1-87c0-d6e6141ddce4","Type":"ContainerDied","Data":"6c7edabbdcf164314867ed13123c666c22fe9da6772ad5f166742ffa91fd7747"}
Jul 27 18:59:35.686552 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:35.686223    2566 scope.go:117] "RemoveContainer" containerID="8af0f2c8066a61b754161be8d4d7d7ac57bceefb8b7f4d16d6f588d7e38585c1"
Jul 27 18:59:35.703068 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:35.703048    2566 scope.go:117] "RemoveContainer" containerID="8af0f2c8066a61b754161be8d4d7d7ac57bceefb8b7f4d16d6f588d7e38585c1"
Jul 27 18:59:35.703445 ip-10-0-128-196 kubenswrapper[2566]: E0727 18:59:35.703417    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8af0f2c8066a61b754161be8d4d7d7ac57bceefb8b7f4d16d6f588d7e38585c1\": container with ID starting with 8af0f2c8066a61b754161be8d4d7d7ac57bceefb8b7f4d16d6f588d7e38585c1 not found: ID does not exist" containerID="8af0f2c8066a61b754161be8d4d7d7ac57bceefb8b7f4d16d6f588d7e38585c1"
Jul 27 18:59:35.703538 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:35.703446    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8af0f2c8066a61b754161be8d4d7d7ac57bceefb8b7f4d16d6f588d7e38585c1"} err="failed to get container status \"8af0f2c8066a61b754161be8d4d7d7ac57bceefb8b7f4d16d6f588d7e38585c1\": rpc error: code = NotFound desc = could not find container \"8af0f2c8066a61b754161be8d4d7d7ac57bceefb8b7f4d16d6f588d7e38585c1\": container with ID starting with 8af0f2c8066a61b754161be8d4d7d7ac57bceefb8b7f4d16d6f588d7e38585c1 not found: ID does not exist"
Jul 27 18:59:35.717728 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:35.717690    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84"]
Jul 27 18:59:35.722599 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:35.722577    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-09eed4cd-859668f6f6-c5q84"]
Jul 27 18:59:36.123716 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:36.123672    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91755248-18ee-49a1-87c0-d6e6141ddce4" path="/var/lib/kubelet/pods/91755248-18ee-49a1-87c0-d6e6141ddce4/volumes"
Jul 27 18:59:43.371595 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:43.371554    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr"]
Jul 27 18:59:43.371952 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:43.371939    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91755248-18ee-49a1-87c0-d6e6141ddce4" containerName="maas-api"
Jul 27 18:59:43.371952 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:43.371950    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="91755248-18ee-49a1-87c0-d6e6141ddce4" containerName="maas-api"
Jul 27 18:59:43.372026 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:43.372016    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="91755248-18ee-49a1-87c0-d6e6141ddce4" containerName="maas-api"
Jul 27 18:59:43.376576 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:43.376558    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr"
Jul 27 18:59:43.381349 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:43.381328    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b9bn6\""
Jul 27 18:59:43.381876 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:43.381863    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-217fd0d6-serving-cert\""
Jul 27 18:59:43.396826 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:43.396804    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr"]
Jul 27 18:59:43.476377 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:43.476347    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60151708-8878-442b-b243-fb57d66dae71-openshift-service-ca\") pod \"maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr\" (UID: \"60151708-8878-442b-b243-fb57d66dae71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr"
Jul 27 18:59:43.476520 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:43.476400    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60151708-8878-442b-b243-fb57d66dae71-maas-api-tls\") pod \"maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr\" (UID: \"60151708-8878-442b-b243-fb57d66dae71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr"
Jul 27 18:59:43.476520 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:43.476444    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkjhx\" (UniqueName: \"kubernetes.io/projected/60151708-8878-442b-b243-fb57d66dae71-kube-api-access-wkjhx\") pod \"maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr\" (UID: \"60151708-8878-442b-b243-fb57d66dae71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr"
Jul 27 18:59:43.577458 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:43.577419    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60151708-8878-442b-b243-fb57d66dae71-openshift-service-ca\") pod \"maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr\" (UID: \"60151708-8878-442b-b243-fb57d66dae71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr"
Jul 27 18:59:43.577632 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:43.577477    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60151708-8878-442b-b243-fb57d66dae71-maas-api-tls\") pod \"maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr\" (UID: \"60151708-8878-442b-b243-fb57d66dae71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr"
Jul 27 18:59:43.577632 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:43.577506    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkjhx\" (UniqueName: \"kubernetes.io/projected/60151708-8878-442b-b243-fb57d66dae71-kube-api-access-wkjhx\") pod \"maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr\" (UID: \"60151708-8878-442b-b243-fb57d66dae71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr"
Jul 27 18:59:43.578051 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:43.578023    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60151708-8878-442b-b243-fb57d66dae71-openshift-service-ca\") pod \"maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr\" (UID: \"60151708-8878-442b-b243-fb57d66dae71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr"
Jul 27 18:59:43.579958 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:43.579935    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60151708-8878-442b-b243-fb57d66dae71-maas-api-tls\") pod \"maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr\" (UID: \"60151708-8878-442b-b243-fb57d66dae71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr"
Jul 27 18:59:43.600540 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:43.600516    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkjhx\" (UniqueName: \"kubernetes.io/projected/60151708-8878-442b-b243-fb57d66dae71-kube-api-access-wkjhx\") pod \"maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr\" (UID: \"60151708-8878-442b-b243-fb57d66dae71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr"
Jul 27 18:59:43.686601 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:43.686513    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr"
Jul 27 18:59:43.837004 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:43.836980    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr"]
Jul 27 18:59:43.838715 ip-10-0-128-196 kubenswrapper[2566]: W0727 18:59:43.838683    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60151708_8878_442b_b243_fb57d66dae71.slice/crio-0ad8b0b22079413e0e0f370389f91373abddbf480bf0db70fc21a57aae31f278 WatchSource:0}: Error finding container 0ad8b0b22079413e0e0f370389f91373abddbf480bf0db70fc21a57aae31f278: Status 404 returned error can't find the container with id 0ad8b0b22079413e0e0f370389f91373abddbf480bf0db70fc21a57aae31f278
Jul 27 18:59:44.737728 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:44.737628    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr" event={"ID":"60151708-8878-442b-b243-fb57d66dae71","Type":"ContainerStarted","Data":"3cfbd8e2f311fa89bd31e4e60db6ff131dbe319d52f272751670ea11453f0b4a"}
Jul 27 18:59:44.737728 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:44.737697    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr" event={"ID":"60151708-8878-442b-b243-fb57d66dae71","Type":"ContainerStarted","Data":"0ad8b0b22079413e0e0f370389f91373abddbf480bf0db70fc21a57aae31f278"}
Jul 27 18:59:44.738274 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:44.737730    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr"
Jul 27 18:59:44.759413 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:44.759361    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr" podStartSLOduration=1.257130213 podStartE2EDuration="1.759346597s" totalImagesPullingTime="502.216384ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:59:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:59:44.758427889 +0000 UTC m=+1845.192685502" watchObservedRunningTime="2026-07-27 18:59:44.759346597 +0000 UTC m=+1845.193604208"
Jul 27 18:59:50.752283 ip-10-0-128-196 kubenswrapper[2566]: I0727 18:59:50.752248    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr"
Jul 27 19:00:19.552678 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:19.552644    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr"]
Jul 27 19:00:19.553163 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:19.552891    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr" podUID="60151708-8878-442b-b243-fb57d66dae71" containerName="maas-api" containerID="cri-o://3cfbd8e2f311fa89bd31e4e60db6ff131dbe319d52f272751670ea11453f0b4a" gracePeriod=30
Jul 27 19:00:19.813445 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:19.813382    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr"
Jul 27 19:00:19.909243 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:19.909211    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkjhx\" (UniqueName: \"kubernetes.io/projected/60151708-8878-442b-b243-fb57d66dae71-kube-api-access-wkjhx\") pod \"60151708-8878-442b-b243-fb57d66dae71\" (UID: \"60151708-8878-442b-b243-fb57d66dae71\") "
Jul 27 19:00:19.909453 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:19.909257    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60151708-8878-442b-b243-fb57d66dae71-maas-api-tls\") pod \"60151708-8878-442b-b243-fb57d66dae71\" (UID: \"60151708-8878-442b-b243-fb57d66dae71\") "
Jul 27 19:00:19.909453 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:19.909380    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60151708-8878-442b-b243-fb57d66dae71-openshift-service-ca\") pod \"60151708-8878-442b-b243-fb57d66dae71\" (UID: \"60151708-8878-442b-b243-fb57d66dae71\") "
Jul 27 19:00:19.909987 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:19.909856    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60151708-8878-442b-b243-fb57d66dae71-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "60151708-8878-442b-b243-fb57d66dae71" (UID: "60151708-8878-442b-b243-fb57d66dae71"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:00:19.911783 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:19.911756    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60151708-8878-442b-b243-fb57d66dae71-kube-api-access-wkjhx" (OuterVolumeSpecName: "kube-api-access-wkjhx") pod "60151708-8878-442b-b243-fb57d66dae71" (UID: "60151708-8878-442b-b243-fb57d66dae71"). InnerVolumeSpecName "kube-api-access-wkjhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:00:19.911783 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:19.911770    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60151708-8878-442b-b243-fb57d66dae71-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "60151708-8878-442b-b243-fb57d66dae71" (UID: "60151708-8878-442b-b243-fb57d66dae71"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:00:19.925371 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:19.925341    2566 generic.go:356] "Generic (PLEG): container finished" podID="60151708-8878-442b-b243-fb57d66dae71" containerID="3cfbd8e2f311fa89bd31e4e60db6ff131dbe319d52f272751670ea11453f0b4a" exitCode=0
Jul 27 19:00:19.925479 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:19.925397    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr"
Jul 27 19:00:19.925479 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:19.925425    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr" event={"ID":"60151708-8878-442b-b243-fb57d66dae71","Type":"ContainerDied","Data":"3cfbd8e2f311fa89bd31e4e60db6ff131dbe319d52f272751670ea11453f0b4a"}
Jul 27 19:00:19.925558 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:19.925495    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr" event={"ID":"60151708-8878-442b-b243-fb57d66dae71","Type":"ContainerDied","Data":"0ad8b0b22079413e0e0f370389f91373abddbf480bf0db70fc21a57aae31f278"}
Jul 27 19:00:19.925558 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:19.925517    2566 scope.go:117] "RemoveContainer" containerID="3cfbd8e2f311fa89bd31e4e60db6ff131dbe319d52f272751670ea11453f0b4a"
Jul 27 19:00:19.934962 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:19.934944    2566 scope.go:117] "RemoveContainer" containerID="3cfbd8e2f311fa89bd31e4e60db6ff131dbe319d52f272751670ea11453f0b4a"
Jul 27 19:00:19.935293 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:00:19.935273    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cfbd8e2f311fa89bd31e4e60db6ff131dbe319d52f272751670ea11453f0b4a\": container with ID starting with 3cfbd8e2f311fa89bd31e4e60db6ff131dbe319d52f272751670ea11453f0b4a not found: ID does not exist" containerID="3cfbd8e2f311fa89bd31e4e60db6ff131dbe319d52f272751670ea11453f0b4a"
Jul 27 19:00:19.935347 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:19.935302    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cfbd8e2f311fa89bd31e4e60db6ff131dbe319d52f272751670ea11453f0b4a"} err="failed to get container status \"3cfbd8e2f311fa89bd31e4e60db6ff131dbe319d52f272751670ea11453f0b4a\": rpc error: code = NotFound desc = could not find container \"3cfbd8e2f311fa89bd31e4e60db6ff131dbe319d52f272751670ea11453f0b4a\": container with ID starting with 3cfbd8e2f311fa89bd31e4e60db6ff131dbe319d52f272751670ea11453f0b4a not found: ID does not exist"
Jul 27 19:00:19.949577 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:19.949551    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr"]
Jul 27 19:00:19.953733 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:19.953709    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-217fd0d6-5cff87495f-l7nzr"]
Jul 27 19:00:20.010025 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:20.009990    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60151708-8878-442b-b243-fb57d66dae71-openshift-service-ca\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:00:20.010025 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:20.010019    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkjhx\" (UniqueName: \"kubernetes.io/projected/60151708-8878-442b-b243-fb57d66dae71-kube-api-access-wkjhx\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:00:20.010025 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:20.010030    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60151708-8878-442b-b243-fb57d66dae71-maas-api-tls\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:00:20.125840 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:20.125716    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60151708-8878-442b-b243-fb57d66dae71" path="/var/lib/kubelet/pods/60151708-8878-442b-b243-fb57d66dae71/volumes"
Jul 27 19:00:28.483900 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:28.483862    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb"]
Jul 27 19:00:28.484473 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:28.484455    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60151708-8878-442b-b243-fb57d66dae71" containerName="maas-api"
Jul 27 19:00:28.484519 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:28.484477    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="60151708-8878-442b-b243-fb57d66dae71" containerName="maas-api"
Jul 27 19:00:28.484557 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:28.484550    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="60151708-8878-442b-b243-fb57d66dae71" containerName="maas-api"
Jul 27 19:00:28.489128 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:28.489096    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb"
Jul 27 19:00:28.492506 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:28.492484    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b9bn6\""
Jul 27 19:00:28.492634 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:28.492552    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-d042bdca-serving-cert\""
Jul 27 19:00:28.501141 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:28.501098    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb"]
Jul 27 19:00:28.586955 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:28.586929    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b51cb917-b48a-447e-9aa8-c6aefb5f19fc-maas-api-tls\") pod \"maas-api-e2e-derive-d042bdca-868b88775d-mg5qb\" (UID: \"b51cb917-b48a-447e-9aa8-c6aefb5f19fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb"
Jul 27 19:00:28.587112 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:28.586979    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b51cb917-b48a-447e-9aa8-c6aefb5f19fc-openshift-service-ca\") pod \"maas-api-e2e-derive-d042bdca-868b88775d-mg5qb\" (UID: \"b51cb917-b48a-447e-9aa8-c6aefb5f19fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb"
Jul 27 19:00:28.587112 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:28.587004    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4hhb\" (UniqueName: \"kubernetes.io/projected/b51cb917-b48a-447e-9aa8-c6aefb5f19fc-kube-api-access-x4hhb\") pod \"maas-api-e2e-derive-d042bdca-868b88775d-mg5qb\" (UID: \"b51cb917-b48a-447e-9aa8-c6aefb5f19fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb"
Jul 27 19:00:28.687919 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:28.687884    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b51cb917-b48a-447e-9aa8-c6aefb5f19fc-openshift-service-ca\") pod \"maas-api-e2e-derive-d042bdca-868b88775d-mg5qb\" (UID: \"b51cb917-b48a-447e-9aa8-c6aefb5f19fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb"
Jul 27 19:00:28.687919 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:28.687922    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4hhb\" (UniqueName: \"kubernetes.io/projected/b51cb917-b48a-447e-9aa8-c6aefb5f19fc-kube-api-access-x4hhb\") pod \"maas-api-e2e-derive-d042bdca-868b88775d-mg5qb\" (UID: \"b51cb917-b48a-447e-9aa8-c6aefb5f19fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb"
Jul 27 19:00:28.688144 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:28.687971    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b51cb917-b48a-447e-9aa8-c6aefb5f19fc-maas-api-tls\") pod \"maas-api-e2e-derive-d042bdca-868b88775d-mg5qb\" (UID: \"b51cb917-b48a-447e-9aa8-c6aefb5f19fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb"
Jul 27 19:00:28.688144 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:00:28.688057    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-serving-cert: secret "maas-api-e2e-derive-d042bdca-serving-cert" not found
Jul 27 19:00:28.688230 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:00:28.688153    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b51cb917-b48a-447e-9aa8-c6aefb5f19fc-maas-api-tls podName:b51cb917-b48a-447e-9aa8-c6aefb5f19fc nodeName:}" failed. No retries permitted until 2026-07-27 19:00:29.188137272 +0000 UTC m=+1889.622394873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b51cb917-b48a-447e-9aa8-c6aefb5f19fc-maas-api-tls") pod "maas-api-e2e-derive-d042bdca-868b88775d-mg5qb" (UID: "b51cb917-b48a-447e-9aa8-c6aefb5f19fc") : secret "maas-api-e2e-derive-d042bdca-serving-cert" not found
Jul 27 19:00:28.688523 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:28.688503    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b51cb917-b48a-447e-9aa8-c6aefb5f19fc-openshift-service-ca\") pod \"maas-api-e2e-derive-d042bdca-868b88775d-mg5qb\" (UID: \"b51cb917-b48a-447e-9aa8-c6aefb5f19fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb"
Jul 27 19:00:28.698435 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:28.698415    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4hhb\" (UniqueName: \"kubernetes.io/projected/b51cb917-b48a-447e-9aa8-c6aefb5f19fc-kube-api-access-x4hhb\") pod \"maas-api-e2e-derive-d042bdca-868b88775d-mg5qb\" (UID: \"b51cb917-b48a-447e-9aa8-c6aefb5f19fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb"
Jul 27 19:00:29.192668 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:29.192639    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b51cb917-b48a-447e-9aa8-c6aefb5f19fc-maas-api-tls\") pod \"maas-api-e2e-derive-d042bdca-868b88775d-mg5qb\" (UID: \"b51cb917-b48a-447e-9aa8-c6aefb5f19fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb"
Jul 27 19:00:29.195484 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:29.195453    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b51cb917-b48a-447e-9aa8-c6aefb5f19fc-maas-api-tls\") pod \"maas-api-e2e-derive-d042bdca-868b88775d-mg5qb\" (UID: \"b51cb917-b48a-447e-9aa8-c6aefb5f19fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb"
Jul 27 19:00:29.398775 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:29.398727    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb"
Jul 27 19:00:29.544011 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:29.543988    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb"]
Jul 27 19:00:29.545264 ip-10-0-128-196 kubenswrapper[2566]: W0727 19:00:29.545239    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb51cb917_b48a_447e_9aa8_c6aefb5f19fc.slice/crio-bb90900b44ef47d1ef65ebc91e5ca1ccf68859192d8862bd184faa1049ca62fc WatchSource:0}: Error finding container bb90900b44ef47d1ef65ebc91e5ca1ccf68859192d8862bd184faa1049ca62fc: Status 404 returned error can't find the container with id bb90900b44ef47d1ef65ebc91e5ca1ccf68859192d8862bd184faa1049ca62fc
Jul 27 19:00:29.980633 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:29.980596    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb" event={"ID":"b51cb917-b48a-447e-9aa8-c6aefb5f19fc","Type":"ContainerStarted","Data":"bb90900b44ef47d1ef65ebc91e5ca1ccf68859192d8862bd184faa1049ca62fc"}
Jul 27 19:00:31.992480 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:31.992437    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb" event={"ID":"b51cb917-b48a-447e-9aa8-c6aefb5f19fc","Type":"ContainerStarted","Data":"d09eaf787dda2af555459682097af2ca75cb6bf1f2535166a6cabb02d97b9785"}
Jul 27 19:00:31.992891 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:31.992537    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb"
Jul 27 19:00:38.005021 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:38.004988    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb"
Jul 27 19:00:38.027232 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:00:38.027170    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb" podStartSLOduration=8.427468104 podStartE2EDuration="10.027155912s" totalImagesPullingTime="1.599687808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:00:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:00:32.021416964 +0000 UTC m=+1892.455674576" watchObservedRunningTime="2026-07-27 19:00:38.027155912 +0000 UTC m=+1898.461413522"
Jul 27 19:01:03.688981 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:01:03.688949    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb"]
Jul 27 19:01:03.689481 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:01:03.689182    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb" podUID="b51cb917-b48a-447e-9aa8-c6aefb5f19fc" containerName="maas-api" containerID="cri-o://d09eaf787dda2af555459682097af2ca75cb6bf1f2535166a6cabb02d97b9785" gracePeriod=30
Jul 27 19:01:03.940197 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:01:03.940141    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb"
Jul 27 19:01:04.002846 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:01:04.002819    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b51cb917-b48a-447e-9aa8-c6aefb5f19fc-openshift-service-ca\") pod \"b51cb917-b48a-447e-9aa8-c6aefb5f19fc\" (UID: \"b51cb917-b48a-447e-9aa8-c6aefb5f19fc\") "
Jul 27 19:01:04.002994 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:01:04.002873    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4hhb\" (UniqueName: \"kubernetes.io/projected/b51cb917-b48a-447e-9aa8-c6aefb5f19fc-kube-api-access-x4hhb\") pod \"b51cb917-b48a-447e-9aa8-c6aefb5f19fc\" (UID: \"b51cb917-b48a-447e-9aa8-c6aefb5f19fc\") "
Jul 27 19:01:04.002994 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:01:04.002913    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b51cb917-b48a-447e-9aa8-c6aefb5f19fc-maas-api-tls\") pod \"b51cb917-b48a-447e-9aa8-c6aefb5f19fc\" (UID: \"b51cb917-b48a-447e-9aa8-c6aefb5f19fc\") "
Jul 27 19:01:04.003253 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:01:04.003227    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b51cb917-b48a-447e-9aa8-c6aefb5f19fc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b51cb917-b48a-447e-9aa8-c6aefb5f19fc" (UID: "b51cb917-b48a-447e-9aa8-c6aefb5f19fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:01:04.005059 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:01:04.005029    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b51cb917-b48a-447e-9aa8-c6aefb5f19fc-kube-api-access-x4hhb" (OuterVolumeSpecName: "kube-api-access-x4hhb") pod "b51cb917-b48a-447e-9aa8-c6aefb5f19fc" (UID: "b51cb917-b48a-447e-9aa8-c6aefb5f19fc"). InnerVolumeSpecName "kube-api-access-x4hhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:01:04.005189 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:01:04.005103    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b51cb917-b48a-447e-9aa8-c6aefb5f19fc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b51cb917-b48a-447e-9aa8-c6aefb5f19fc" (UID: "b51cb917-b48a-447e-9aa8-c6aefb5f19fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:01:04.103811 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:01:04.103785    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4hhb\" (UniqueName: \"kubernetes.io/projected/b51cb917-b48a-447e-9aa8-c6aefb5f19fc-kube-api-access-x4hhb\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:01:04.103811 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:01:04.103807    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b51cb917-b48a-447e-9aa8-c6aefb5f19fc-maas-api-tls\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:01:04.103811 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:01:04.103817    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b51cb917-b48a-447e-9aa8-c6aefb5f19fc-openshift-service-ca\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:01:04.163104 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:01:04.163075    2566 generic.go:356] "Generic (PLEG): container finished" podID="b51cb917-b48a-447e-9aa8-c6aefb5f19fc" containerID="d09eaf787dda2af555459682097af2ca75cb6bf1f2535166a6cabb02d97b9785" exitCode=0
Jul 27 19:01:04.163290 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:01:04.163145    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb"
Jul 27 19:01:04.163290 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:01:04.163164    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb" event={"ID":"b51cb917-b48a-447e-9aa8-c6aefb5f19fc","Type":"ContainerDied","Data":"d09eaf787dda2af555459682097af2ca75cb6bf1f2535166a6cabb02d97b9785"}
Jul 27 19:01:04.163290 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:01:04.163224    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb" event={"ID":"b51cb917-b48a-447e-9aa8-c6aefb5f19fc","Type":"ContainerDied","Data":"bb90900b44ef47d1ef65ebc91e5ca1ccf68859192d8862bd184faa1049ca62fc"}
Jul 27 19:01:04.163290 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:01:04.163255    2566 scope.go:117] "RemoveContainer" containerID="d09eaf787dda2af555459682097af2ca75cb6bf1f2535166a6cabb02d97b9785"
Jul 27 19:01:04.172475 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:01:04.172461    2566 scope.go:117] "RemoveContainer" containerID="d09eaf787dda2af555459682097af2ca75cb6bf1f2535166a6cabb02d97b9785"
Jul 27 19:01:04.172700 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:01:04.172680    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d09eaf787dda2af555459682097af2ca75cb6bf1f2535166a6cabb02d97b9785\": container with ID starting with d09eaf787dda2af555459682097af2ca75cb6bf1f2535166a6cabb02d97b9785 not found: ID does not exist" containerID="d09eaf787dda2af555459682097af2ca75cb6bf1f2535166a6cabb02d97b9785"
Jul 27 19:01:04.172751 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:01:04.172709    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d09eaf787dda2af555459682097af2ca75cb6bf1f2535166a6cabb02d97b9785"} err="failed to get container status \"d09eaf787dda2af555459682097af2ca75cb6bf1f2535166a6cabb02d97b9785\": rpc error: code = NotFound desc = could not find container \"d09eaf787dda2af555459682097af2ca75cb6bf1f2535166a6cabb02d97b9785\": container with ID starting with d09eaf787dda2af555459682097af2ca75cb6bf1f2535166a6cabb02d97b9785 not found: ID does not exist"
Jul 27 19:01:04.189210 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:01:04.189179    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb"]
Jul 27 19:01:04.189334 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:01:04.189232    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d042bdca-868b88775d-mg5qb"]
Jul 27 19:01:06.124576 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:01:06.124538    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b51cb917-b48a-447e-9aa8-c6aefb5f19fc" path="/var/lib/kubelet/pods/b51cb917-b48a-447e-9aa8-c6aefb5f19fc/volumes"
Jul 27 19:02:51.728268 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:02:51.728239    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx"]
Jul 27 19:02:51.728717 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:02:51.728645    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b51cb917-b48a-447e-9aa8-c6aefb5f19fc" containerName="maas-api"
Jul 27 19:02:51.728717 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:02:51.728656    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b51cb917-b48a-447e-9aa8-c6aefb5f19fc" containerName="maas-api"
Jul 27 19:02:51.728717 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:02:51.728716    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b51cb917-b48a-447e-9aa8-c6aefb5f19fc" containerName="maas-api"
Jul 27 19:02:51.737727 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:02:51.737700    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx"
Jul 27 19:02:51.741206 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:02:51.741176    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9aed4c03-serving-cert\""
Jul 27 19:02:51.741688 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:02:51.741666    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b9bn6\""
Jul 27 19:02:51.742226 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:02:51.742188    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx"]
Jul 27 19:02:51.854429 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:02:51.854395    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c786769-4c72-4e70-b7e0-319b2b7d8e38-openshift-service-ca\") pod \"maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx\" (UID: \"0c786769-4c72-4e70-b7e0-319b2b7d8e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx"
Jul 27 19:02:51.854429 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:02:51.854427    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c786769-4c72-4e70-b7e0-319b2b7d8e38-maas-api-tls\") pod \"maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx\" (UID: \"0c786769-4c72-4e70-b7e0-319b2b7d8e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx"
Jul 27 19:02:51.854650 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:02:51.854471    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6p7w\" (UniqueName: \"kubernetes.io/projected/0c786769-4c72-4e70-b7e0-319b2b7d8e38-kube-api-access-g6p7w\") pod \"maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx\" (UID: \"0c786769-4c72-4e70-b7e0-319b2b7d8e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx"
Jul 27 19:02:51.955355 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:02:51.955319    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6p7w\" (UniqueName: \"kubernetes.io/projected/0c786769-4c72-4e70-b7e0-319b2b7d8e38-kube-api-access-g6p7w\") pod \"maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx\" (UID: \"0c786769-4c72-4e70-b7e0-319b2b7d8e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx"
Jul 27 19:02:51.955538 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:02:51.955391    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c786769-4c72-4e70-b7e0-319b2b7d8e38-openshift-service-ca\") pod \"maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx\" (UID: \"0c786769-4c72-4e70-b7e0-319b2b7d8e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx"
Jul 27 19:02:51.955538 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:02:51.955409    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c786769-4c72-4e70-b7e0-319b2b7d8e38-maas-api-tls\") pod \"maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx\" (UID: \"0c786769-4c72-4e70-b7e0-319b2b7d8e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx"
Jul 27 19:02:51.956072 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:02:51.956051    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c786769-4c72-4e70-b7e0-319b2b7d8e38-openshift-service-ca\") pod \"maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx\" (UID: \"0c786769-4c72-4e70-b7e0-319b2b7d8e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx"
Jul 27 19:02:51.957910 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:02:51.957893    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c786769-4c72-4e70-b7e0-319b2b7d8e38-maas-api-tls\") pod \"maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx\" (UID: \"0c786769-4c72-4e70-b7e0-319b2b7d8e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx"
Jul 27 19:02:51.964484 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:02:51.964453    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6p7w\" (UniqueName: \"kubernetes.io/projected/0c786769-4c72-4e70-b7e0-319b2b7d8e38-kube-api-access-g6p7w\") pod \"maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx\" (UID: \"0c786769-4c72-4e70-b7e0-319b2b7d8e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx"
Jul 27 19:02:52.050273 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:02:52.050248    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx"
Jul 27 19:02:52.193732 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:02:52.193698    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx"]
Jul 27 19:02:52.197080 ip-10-0-128-196 kubenswrapper[2566]: W0727 19:02:52.197052    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c786769_4c72_4e70_b7e0_319b2b7d8e38.slice/crio-eeee567f944fdb1122b83a3546d727d9fe78faece5088eecf794f987eb508d7a WatchSource:0}: Error finding container eeee567f944fdb1122b83a3546d727d9fe78faece5088eecf794f987eb508d7a: Status 404 returned error can't find the container with id eeee567f944fdb1122b83a3546d727d9fe78faece5088eecf794f987eb508d7a
Jul 27 19:02:52.762271 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:02:52.762233    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx" event={"ID":"0c786769-4c72-4e70-b7e0-319b2b7d8e38","Type":"ContainerStarted","Data":"eeee567f944fdb1122b83a3546d727d9fe78faece5088eecf794f987eb508d7a"}
Jul 27 19:02:54.774223 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:02:54.774188    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx" event={"ID":"0c786769-4c72-4e70-b7e0-319b2b7d8e38","Type":"ContainerStarted","Data":"fcc213e4578f653dcfd04ba189725d0cfc910c909dc0db9ca92efb014429de13"}
Jul 27 19:02:54.774673 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:02:54.774318    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx"
Jul 27 19:02:54.795925 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:02:54.795878    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx" podStartSLOduration=2.057261399 podStartE2EDuration="3.795864778s" totalImagesPullingTime="1.738603379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:02:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:02:54.792914946 +0000 UTC m=+2035.227172560" watchObservedRunningTime="2026-07-27 19:02:54.795864778 +0000 UTC m=+2035.230122400"
Jul 27 19:03:00.785612 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:00.785585    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx"
Jul 27 19:03:14.462401 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:14.462362    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx"]
Jul 27 19:03:14.462807 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:14.462595    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx" podUID="0c786769-4c72-4e70-b7e0-319b2b7d8e38" containerName="maas-api" containerID="cri-o://fcc213e4578f653dcfd04ba189725d0cfc910c909dc0db9ca92efb014429de13" gracePeriod=30
Jul 27 19:03:14.715817 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:14.715758    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx"
Jul 27 19:03:14.853250 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:14.853216    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6p7w\" (UniqueName: \"kubernetes.io/projected/0c786769-4c72-4e70-b7e0-319b2b7d8e38-kube-api-access-g6p7w\") pod \"0c786769-4c72-4e70-b7e0-319b2b7d8e38\" (UID: \"0c786769-4c72-4e70-b7e0-319b2b7d8e38\") "
Jul 27 19:03:14.853425 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:14.853329    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c786769-4c72-4e70-b7e0-319b2b7d8e38-maas-api-tls\") pod \"0c786769-4c72-4e70-b7e0-319b2b7d8e38\" (UID: \"0c786769-4c72-4e70-b7e0-319b2b7d8e38\") "
Jul 27 19:03:14.853425 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:14.853370    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c786769-4c72-4e70-b7e0-319b2b7d8e38-openshift-service-ca\") pod \"0c786769-4c72-4e70-b7e0-319b2b7d8e38\" (UID: \"0c786769-4c72-4e70-b7e0-319b2b7d8e38\") "
Jul 27 19:03:14.853751 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:14.853724    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c786769-4c72-4e70-b7e0-319b2b7d8e38-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0c786769-4c72-4e70-b7e0-319b2b7d8e38" (UID: "0c786769-4c72-4e70-b7e0-319b2b7d8e38"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:03:14.855505 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:14.855485    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c786769-4c72-4e70-b7e0-319b2b7d8e38-kube-api-access-g6p7w" (OuterVolumeSpecName: "kube-api-access-g6p7w") pod "0c786769-4c72-4e70-b7e0-319b2b7d8e38" (UID: "0c786769-4c72-4e70-b7e0-319b2b7d8e38"). InnerVolumeSpecName "kube-api-access-g6p7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:03:14.855505 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:14.855493    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c786769-4c72-4e70-b7e0-319b2b7d8e38-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0c786769-4c72-4e70-b7e0-319b2b7d8e38" (UID: "0c786769-4c72-4e70-b7e0-319b2b7d8e38"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:03:14.885753 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:14.885719    2566 generic.go:356] "Generic (PLEG): container finished" podID="0c786769-4c72-4e70-b7e0-319b2b7d8e38" containerID="fcc213e4578f653dcfd04ba189725d0cfc910c909dc0db9ca92efb014429de13" exitCode=0
Jul 27 19:03:14.885906 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:14.885764    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx"
Jul 27 19:03:14.885906 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:14.885803    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx" event={"ID":"0c786769-4c72-4e70-b7e0-319b2b7d8e38","Type":"ContainerDied","Data":"fcc213e4578f653dcfd04ba189725d0cfc910c909dc0db9ca92efb014429de13"}
Jul 27 19:03:14.885906 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:14.885854    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx" event={"ID":"0c786769-4c72-4e70-b7e0-319b2b7d8e38","Type":"ContainerDied","Data":"eeee567f944fdb1122b83a3546d727d9fe78faece5088eecf794f987eb508d7a"}
Jul 27 19:03:14.885906 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:14.885867    2566 scope.go:117] "RemoveContainer" containerID="fcc213e4578f653dcfd04ba189725d0cfc910c909dc0db9ca92efb014429de13"
Jul 27 19:03:14.895301 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:14.895285    2566 scope.go:117] "RemoveContainer" containerID="fcc213e4578f653dcfd04ba189725d0cfc910c909dc0db9ca92efb014429de13"
Jul 27 19:03:14.895558 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:03:14.895537    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcc213e4578f653dcfd04ba189725d0cfc910c909dc0db9ca92efb014429de13\": container with ID starting with fcc213e4578f653dcfd04ba189725d0cfc910c909dc0db9ca92efb014429de13 not found: ID does not exist" containerID="fcc213e4578f653dcfd04ba189725d0cfc910c909dc0db9ca92efb014429de13"
Jul 27 19:03:14.895649 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:14.895563    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc213e4578f653dcfd04ba189725d0cfc910c909dc0db9ca92efb014429de13"} err="failed to get container status \"fcc213e4578f653dcfd04ba189725d0cfc910c909dc0db9ca92efb014429de13\": rpc error: code = NotFound desc = could not find container \"fcc213e4578f653dcfd04ba189725d0cfc910c909dc0db9ca92efb014429de13\": container with ID starting with fcc213e4578f653dcfd04ba189725d0cfc910c909dc0db9ca92efb014429de13 not found: ID does not exist"
Jul 27 19:03:14.907762 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:14.907740    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx"]
Jul 27 19:03:14.912296 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:14.912275    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9aed4c03-6cdc5669d9-9rqfx"]
Jul 27 19:03:14.954909 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:14.954885    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c786769-4c72-4e70-b7e0-319b2b7d8e38-maas-api-tls\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:03:14.954909 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:14.954905    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c786769-4c72-4e70-b7e0-319b2b7d8e38-openshift-service-ca\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:03:14.955042 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:14.954915    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6p7w\" (UniqueName: \"kubernetes.io/projected/0c786769-4c72-4e70-b7e0-319b2b7d8e38-kube-api-access-g6p7w\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:03:16.123436 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:16.123406    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c786769-4c72-4e70-b7e0-319b2b7d8e38" path="/var/lib/kubelet/pods/0c786769-4c72-4e70-b7e0-319b2b7d8e38/volumes"
Jul 27 19:03:49.972980 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:49.972912    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x"]
Jul 27 19:03:49.973476 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:49.973390    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c786769-4c72-4e70-b7e0-319b2b7d8e38" containerName="maas-api"
Jul 27 19:03:49.973476 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:49.973404    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c786769-4c72-4e70-b7e0-319b2b7d8e38" containerName="maas-api"
Jul 27 19:03:49.973607 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:49.973489    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c786769-4c72-4e70-b7e0-319b2b7d8e38" containerName="maas-api"
Jul 27 19:03:49.976408 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:49.976391    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x"
Jul 27 19:03:49.979441 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:49.979421    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fc9cd50f-serving-cert\""
Jul 27 19:03:49.979558 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:49.979469    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b9bn6\""
Jul 27 19:03:49.987978 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:49.987956    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x"]
Jul 27 19:03:50.063098 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:50.063069    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpzqv\" (UniqueName: \"kubernetes.io/projected/557e14b3-4935-42ea-b5f7-c7eb6e643436-kube-api-access-jpzqv\") pod \"maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x\" (UID: \"557e14b3-4935-42ea-b5f7-c7eb6e643436\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x"
Jul 27 19:03:50.063262 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:50.063107    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/557e14b3-4935-42ea-b5f7-c7eb6e643436-maas-api-tls\") pod \"maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x\" (UID: \"557e14b3-4935-42ea-b5f7-c7eb6e643436\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x"
Jul 27 19:03:50.063262 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:50.063235    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/557e14b3-4935-42ea-b5f7-c7eb6e643436-openshift-service-ca\") pod \"maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x\" (UID: \"557e14b3-4935-42ea-b5f7-c7eb6e643436\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x"
Jul 27 19:03:50.163943 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:50.163910    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/557e14b3-4935-42ea-b5f7-c7eb6e643436-openshift-service-ca\") pod \"maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x\" (UID: \"557e14b3-4935-42ea-b5f7-c7eb6e643436\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x"
Jul 27 19:03:50.164182 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:50.163959    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpzqv\" (UniqueName: \"kubernetes.io/projected/557e14b3-4935-42ea-b5f7-c7eb6e643436-kube-api-access-jpzqv\") pod \"maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x\" (UID: \"557e14b3-4935-42ea-b5f7-c7eb6e643436\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x"
Jul 27 19:03:50.164182 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:50.163988    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/557e14b3-4935-42ea-b5f7-c7eb6e643436-maas-api-tls\") pod \"maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x\" (UID: \"557e14b3-4935-42ea-b5f7-c7eb6e643436\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x"
Jul 27 19:03:50.164182 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:03:50.164149    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-serving-cert: secret "maas-api-e2e-mt-fc9cd50f-serving-cert" not found
Jul 27 19:03:50.164382 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:03:50.164230    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/557e14b3-4935-42ea-b5f7-c7eb6e643436-maas-api-tls podName:557e14b3-4935-42ea-b5f7-c7eb6e643436 nodeName:}" failed. No retries permitted until 2026-07-27 19:03:50.664212651 +0000 UTC m=+2091.098470263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/557e14b3-4935-42ea-b5f7-c7eb6e643436-maas-api-tls") pod "maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x" (UID: "557e14b3-4935-42ea-b5f7-c7eb6e643436") : secret "maas-api-e2e-mt-fc9cd50f-serving-cert" not found
Jul 27 19:03:50.164685 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:50.164660    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/557e14b3-4935-42ea-b5f7-c7eb6e643436-openshift-service-ca\") pod \"maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x\" (UID: \"557e14b3-4935-42ea-b5f7-c7eb6e643436\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x"
Jul 27 19:03:50.177196 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:50.176788    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpzqv\" (UniqueName: \"kubernetes.io/projected/557e14b3-4935-42ea-b5f7-c7eb6e643436-kube-api-access-jpzqv\") pod \"maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x\" (UID: \"557e14b3-4935-42ea-b5f7-c7eb6e643436\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x"
Jul 27 19:03:50.668921 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:50.668878    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/557e14b3-4935-42ea-b5f7-c7eb6e643436-maas-api-tls\") pod \"maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x\" (UID: \"557e14b3-4935-42ea-b5f7-c7eb6e643436\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x"
Jul 27 19:03:50.671403 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:50.671379    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/557e14b3-4935-42ea-b5f7-c7eb6e643436-maas-api-tls\") pod \"maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x\" (UID: \"557e14b3-4935-42ea-b5f7-c7eb6e643436\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x"
Jul 27 19:03:50.887093 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:50.887050    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x"
Jul 27 19:03:51.026628 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:51.026605    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x"]
Jul 27 19:03:51.028274 ip-10-0-128-196 kubenswrapper[2566]: W0727 19:03:51.028249    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod557e14b3_4935_42ea_b5f7_c7eb6e643436.slice/crio-19d20969ee1e7cdc232a0098005bceaf7123237675734ca9580057c2274ad4d7 WatchSource:0}: Error finding container 19d20969ee1e7cdc232a0098005bceaf7123237675734ca9580057c2274ad4d7: Status 404 returned error can't find the container with id 19d20969ee1e7cdc232a0098005bceaf7123237675734ca9580057c2274ad4d7
Jul 27 19:03:51.029508 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:51.029491    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 19:03:51.076521 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:51.076489    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x" event={"ID":"557e14b3-4935-42ea-b5f7-c7eb6e643436","Type":"ContainerStarted","Data":"19d20969ee1e7cdc232a0098005bceaf7123237675734ca9580057c2274ad4d7"}
Jul 27 19:03:52.083866 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:52.083819    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x" event={"ID":"557e14b3-4935-42ea-b5f7-c7eb6e643436","Type":"ContainerStarted","Data":"bf93ee34b64bcb8668df190b022d993d34c5eb872953656e76fe46129ae5f5f0"}
Jul 27 19:03:52.084359 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:52.083890    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x"
Jul 27 19:03:52.102727 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:52.102678    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x" podStartSLOduration=2.681581251 podStartE2EDuration="3.102662641s" totalImagesPullingTime="421.08139ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:03:52.101416498 +0000 UTC m=+2092.535674121" watchObservedRunningTime="2026-07-27 19:03:52.102662641 +0000 UTC m=+2092.536920306"
Jul 27 19:03:58.095266 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:03:58.095236    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x"
Jul 27 19:04:00.212880 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:00.212856    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 19:04:00.223346 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:00.223326    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 19:04:04.487006 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:04.486968    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"]
Jul 27 19:04:04.490550 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:04.490533    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"
Jul 27 19:04:04.494925 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:04.494893    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fc9cd50f\"/\"kube-root-ca.crt\""
Jul 27 19:04:04.494925 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:04.494894    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fc9cd50f\"/\"e2e-lifecycle-model-fc9cd50f-kserve-self-signed-certs\""
Jul 27 19:04:04.495159 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:04.494954    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fc9cd50f\"/\"default-dockercfg-zc4qs\""
Jul 27 19:04:04.495159 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:04.494960    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fc9cd50f\"/\"openshift-service-ca.crt\""
Jul 27 19:04:04.500747 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:04.500725    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"]
Jul 27 19:04:04.606727 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:04.606690    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-dshm\") pod \"e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8\" (UID: \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\") " pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"
Jul 27 19:04:04.606727 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:04.606730    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-tls-certs\") pod \"e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8\" (UID: \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\") " pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"
Jul 27 19:04:04.606962 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:04.606749    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-home\") pod \"e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8\" (UID: \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\") " pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"
Jul 27 19:04:04.606962 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:04.606872    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-tmp-dir\") pod \"e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8\" (UID: \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\") " pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"
Jul 27 19:04:04.606962 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:04.606904    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-model-cache\") pod \"e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8\" (UID: \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\") " pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"
Jul 27 19:04:04.606962 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:04.606926    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gm6n\" (UniqueName: \"kubernetes.io/projected/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-kube-api-access-9gm6n\") pod \"e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8\" (UID: \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\") " pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"
Jul 27 19:04:04.708025 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:04.707987    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-dshm\") pod \"e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8\" (UID: \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\") " pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"
Jul 27 19:04:04.708025 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:04.708027    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-tls-certs\") pod \"e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8\" (UID: \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\") " pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"
Jul 27 19:04:04.708339 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:04.708053    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-home\") pod \"e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8\" (UID: \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\") " pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"
Jul 27 19:04:04.708339 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:04.708156    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-tmp-dir\") pod \"e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8\" (UID: \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\") " pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"
Jul 27 19:04:04.708339 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:04.708184    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-model-cache\") pod \"e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8\" (UID: \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\") " pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"
Jul 27 19:04:04.708339 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:04.708217    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gm6n\" (UniqueName: \"kubernetes.io/projected/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-kube-api-access-9gm6n\") pod \"e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8\" (UID: \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\") " pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"
Jul 27 19:04:04.708536 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:04.708503    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-home\") pod \"e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8\" (UID: \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\") " pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"
Jul 27 19:04:04.708589 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:04.708579    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-tmp-dir\") pod \"e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8\" (UID: \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\") " pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"
Jul 27 19:04:04.708634 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:04.708573    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-model-cache\") pod \"e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8\" (UID: \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\") " pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"
Jul 27 19:04:04.710524 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:04.710501    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-dshm\") pod \"e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8\" (UID: \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\") " pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"
Jul 27 19:04:04.710802 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:04.710782    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-tls-certs\") pod \"e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8\" (UID: \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\") " pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"
Jul 27 19:04:04.718218 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:04.718190    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gm6n\" (UniqueName: \"kubernetes.io/projected/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-kube-api-access-9gm6n\") pod \"e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8\" (UID: \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\") " pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"
Jul 27 19:04:04.801044 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:04.801002    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"
Jul 27 19:04:04.940042 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:04.940016    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"]
Jul 27 19:04:04.941429 ip-10-0-128-196 kubenswrapper[2566]: W0727 19:04:04.941402    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0c33ec2_cbaf_4fb3_a9a6_3693e7b91948.slice/crio-91e217bde894966bd8701be35a56a0de0e23fb6ad6b2a582daa8032270e0a9eb WatchSource:0}: Error finding container 91e217bde894966bd8701be35a56a0de0e23fb6ad6b2a582daa8032270e0a9eb: Status 404 returned error can't find the container with id 91e217bde894966bd8701be35a56a0de0e23fb6ad6b2a582daa8032270e0a9eb
Jul 27 19:04:05.154344 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:05.154252    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8" event={"ID":"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948","Type":"ContainerStarted","Data":"9c05e97f892ae2f415bfd773ba7e52393916d78089bd6bd66f9d3987c9e3ba9c"}
Jul 27 19:04:05.154344 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:05.154299    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8" event={"ID":"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948","Type":"ContainerStarted","Data":"91e217bde894966bd8701be35a56a0de0e23fb6ad6b2a582daa8032270e0a9eb"}
Jul 27 19:04:05.175002 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:05.174957    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8" podStartSLOduration=1.174943157 podStartE2EDuration="1.174943157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:04:05.1735508 +0000 UTC m=+2105.607808423" watchObservedRunningTime="2026-07-27 19:04:05.174943157 +0000 UTC m=+2105.609200767"
Jul 27 19:04:14.801439 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:14.801398    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"
Jul 27 19:04:14.801863 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:14.801455    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"
Jul 27 19:04:14.814041 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:14.814013    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"
Jul 27 19:04:15.217604 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:04:15.217525    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"
Jul 27 19:07:43.193264 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:07:43.193228    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x"]
Jul 27 19:07:43.193820 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:07:43.193485    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x" podUID="557e14b3-4935-42ea-b5f7-c7eb6e643436" containerName="maas-api" containerID="cri-o://bf93ee34b64bcb8668df190b022d993d34c5eb872953656e76fe46129ae5f5f0" gracePeriod=30
Jul 27 19:07:43.317171 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:07:43.317138    2566 generic.go:356] "Generic (PLEG): container finished" podID="557e14b3-4935-42ea-b5f7-c7eb6e643436" containerID="bf93ee34b64bcb8668df190b022d993d34c5eb872953656e76fe46129ae5f5f0" exitCode=0
Jul 27 19:07:43.317171 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:07:43.317155    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x" event={"ID":"557e14b3-4935-42ea-b5f7-c7eb6e643436","Type":"ContainerDied","Data":"bf93ee34b64bcb8668df190b022d993d34c5eb872953656e76fe46129ae5f5f0"}
Jul 27 19:07:43.450739 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:07:43.450676    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x"
Jul 27 19:07:43.582969 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:07:43.582939    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/557e14b3-4935-42ea-b5f7-c7eb6e643436-openshift-service-ca\") pod \"557e14b3-4935-42ea-b5f7-c7eb6e643436\" (UID: \"557e14b3-4935-42ea-b5f7-c7eb6e643436\") "
Jul 27 19:07:43.583137 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:07:43.582996    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpzqv\" (UniqueName: \"kubernetes.io/projected/557e14b3-4935-42ea-b5f7-c7eb6e643436-kube-api-access-jpzqv\") pod \"557e14b3-4935-42ea-b5f7-c7eb6e643436\" (UID: \"557e14b3-4935-42ea-b5f7-c7eb6e643436\") "
Jul 27 19:07:43.583137 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:07:43.583027    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/557e14b3-4935-42ea-b5f7-c7eb6e643436-maas-api-tls\") pod \"557e14b3-4935-42ea-b5f7-c7eb6e643436\" (UID: \"557e14b3-4935-42ea-b5f7-c7eb6e643436\") "
Jul 27 19:07:43.583367 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:07:43.583337    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/557e14b3-4935-42ea-b5f7-c7eb6e643436-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "557e14b3-4935-42ea-b5f7-c7eb6e643436" (UID: "557e14b3-4935-42ea-b5f7-c7eb6e643436"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:07:43.585328 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:07:43.585300    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/557e14b3-4935-42ea-b5f7-c7eb6e643436-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "557e14b3-4935-42ea-b5f7-c7eb6e643436" (UID: "557e14b3-4935-42ea-b5f7-c7eb6e643436"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:07:43.585449 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:07:43.585309    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/557e14b3-4935-42ea-b5f7-c7eb6e643436-kube-api-access-jpzqv" (OuterVolumeSpecName: "kube-api-access-jpzqv") pod "557e14b3-4935-42ea-b5f7-c7eb6e643436" (UID: "557e14b3-4935-42ea-b5f7-c7eb6e643436"). InnerVolumeSpecName "kube-api-access-jpzqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:07:43.684512 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:07:43.684474    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/557e14b3-4935-42ea-b5f7-c7eb6e643436-openshift-service-ca\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:07:43.684512 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:07:43.684506    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpzqv\" (UniqueName: \"kubernetes.io/projected/557e14b3-4935-42ea-b5f7-c7eb6e643436-kube-api-access-jpzqv\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:07:43.684512 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:07:43.684517    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/557e14b3-4935-42ea-b5f7-c7eb6e643436-maas-api-tls\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:07:44.324250 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:07:44.324220    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x"
Jul 27 19:07:44.324250 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:07:44.324241    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x" event={"ID":"557e14b3-4935-42ea-b5f7-c7eb6e643436","Type":"ContainerDied","Data":"19d20969ee1e7cdc232a0098005bceaf7123237675734ca9580057c2274ad4d7"}
Jul 27 19:07:44.324710 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:07:44.324298    2566 scope.go:117] "RemoveContainer" containerID="bf93ee34b64bcb8668df190b022d993d34c5eb872953656e76fe46129ae5f5f0"
Jul 27 19:07:44.357508 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:07:44.357479    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x"]
Jul 27 19:07:44.363312 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:07:44.363285    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fc9cd50f-6fcff5bffd-7z22x"]
Jul 27 19:07:46.123842 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:07:46.123803    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="557e14b3-4935-42ea-b5f7-c7eb6e643436" path="/var/lib/kubelet/pods/557e14b3-4935-42ea-b5f7-c7eb6e643436/volumes"
Jul 27 19:07:53.215530 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:07:53.215490    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"]
Jul 27 19:08:08.235335 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:08.235278    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8" podUID="a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948" containerName="main" containerID="cri-o://9c05e97f892ae2f415bfd773ba7e52393916d78089bd6bd66f9d3987c9e3ba9c" gracePeriod=45
Jul 27 19:08:08.452847 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:08.452812    2566 generic.go:356] "Generic (PLEG): container finished" podID="a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948" containerID="9c05e97f892ae2f415bfd773ba7e52393916d78089bd6bd66f9d3987c9e3ba9c" exitCode=0
Jul 27 19:08:08.452990 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:08.452902    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8" event={"ID":"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948","Type":"ContainerDied","Data":"9c05e97f892ae2f415bfd773ba7e52393916d78089bd6bd66f9d3987c9e3ba9c"}
Jul 27 19:08:08.490322 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:08.490257    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"
Jul 27 19:08:08.622937 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:08.622902    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-tmp-dir\") pod \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\" (UID: \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\") "
Jul 27 19:08:08.623154 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:08.622959    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-home\") pod \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\" (UID: \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\") "
Jul 27 19:08:08.623154 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:08.623003    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-tls-certs\") pod \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\" (UID: \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\") "
Jul 27 19:08:08.623282 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:08.623081    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-model-cache\") pod \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\" (UID: \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\") "
Jul 27 19:08:08.623365 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:08.623343    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gm6n\" (UniqueName: \"kubernetes.io/projected/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-kube-api-access-9gm6n\") pod \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\" (UID: \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\") "
Jul 27 19:08:08.623555 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:08.623390    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948" (UID: "a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:08:08.623647 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:08.623466    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-model-cache" (OuterVolumeSpecName: "model-cache") pod "a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948" (UID: "a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:08:08.623647 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:08.623540    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-dshm\") pod \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\" (UID: \"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948\") "
Jul 27 19:08:08.623647 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:08.623479    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-home" (OuterVolumeSpecName: "home") pod "a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948" (UID: "a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:08:08.623837 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:08.623820    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-model-cache\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:08:08.624064 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:08.623843    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-tmp-dir\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:08:08.624064 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:08.623857    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-home\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:08:08.625641 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:08.625610    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-dshm" (OuterVolumeSpecName: "dshm") pod "a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948" (UID: "a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:08:08.625641 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:08.625618    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-kube-api-access-9gm6n" (OuterVolumeSpecName: "kube-api-access-9gm6n") pod "a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948" (UID: "a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948"). InnerVolumeSpecName "kube-api-access-9gm6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:08:08.625641 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:08.625620    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948" (UID: "a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:08:08.724698 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:08.724663    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-tls-certs\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:08:08.724698 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:08.724695    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gm6n\" (UniqueName: \"kubernetes.io/projected/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-kube-api-access-9gm6n\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:08:08.724865 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:08.724710    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948-dshm\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:08:09.460698 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:09.460668    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"
Jul 27 19:08:09.461188 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:09.460693    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8" event={"ID":"a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948","Type":"ContainerDied","Data":"91e217bde894966bd8701be35a56a0de0e23fb6ad6b2a582daa8032270e0a9eb"}
Jul 27 19:08:09.461188 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:09.460748    2566 scope.go:117] "RemoveContainer" containerID="9c05e97f892ae2f415bfd773ba7e52393916d78089bd6bd66f9d3987c9e3ba9c"
Jul 27 19:08:09.491186 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:09.491161    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"]
Jul 27 19:08:09.493374 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:09.493354    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-fc9cd50f/e2e-lifecycle-model-fc9cd50f-kserve-5cf888857f-cxkq8"]
Jul 27 19:08:10.125391 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:08:10.125357    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948" path="/var/lib/kubelet/pods/a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948/volumes"
Jul 27 19:09:00.249285 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:00.249259    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 19:09:00.261785 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:00.261763    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 19:09:48.899293 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:48.899214    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz"]
Jul 27 19:09:48.899743 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:48.899586    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="557e14b3-4935-42ea-b5f7-c7eb6e643436" containerName="maas-api"
Jul 27 19:09:48.899743 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:48.899596    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="557e14b3-4935-42ea-b5f7-c7eb6e643436" containerName="maas-api"
Jul 27 19:09:48.899743 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:48.899613    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948" containerName="main"
Jul 27 19:09:48.899743 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:48.899618    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948" containerName="main"
Jul 27 19:09:48.899743 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:48.899688    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0c33ec2-cbaf-4fb3-a9a6-3693e7b91948" containerName="main"
Jul 27 19:09:48.899743 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:48.899700    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="557e14b3-4935-42ea-b5f7-c7eb6e643436" containerName="maas-api"
Jul 27 19:09:48.902736 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:48.902720    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz"
Jul 27 19:09:48.905853 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:48.905833    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f9caec-serving-cert\""
Jul 27 19:09:48.905968 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:48.905864    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b9bn6\""
Jul 27 19:09:48.911489 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:48.911466    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz"]
Jul 27 19:09:48.915714 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:48.915689    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hnsn\" (UniqueName: \"kubernetes.io/projected/926b47a9-b1d0-4987-ad8c-9bacb98ff651-kube-api-access-5hnsn\") pod \"maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz\" (UID: \"926b47a9-b1d0-4987-ad8c-9bacb98ff651\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz"
Jul 27 19:09:48.915807 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:48.915735    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/926b47a9-b1d0-4987-ad8c-9bacb98ff651-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz\" (UID: \"926b47a9-b1d0-4987-ad8c-9bacb98ff651\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz"
Jul 27 19:09:48.915867 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:48.915821    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/926b47a9-b1d0-4987-ad8c-9bacb98ff651-maas-api-tls\") pod \"maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz\" (UID: \"926b47a9-b1d0-4987-ad8c-9bacb98ff651\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz"
Jul 27 19:09:49.016992 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:49.016961    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/926b47a9-b1d0-4987-ad8c-9bacb98ff651-maas-api-tls\") pod \"maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz\" (UID: \"926b47a9-b1d0-4987-ad8c-9bacb98ff651\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz"
Jul 27 19:09:49.017200 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:49.017015    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hnsn\" (UniqueName: \"kubernetes.io/projected/926b47a9-b1d0-4987-ad8c-9bacb98ff651-kube-api-access-5hnsn\") pod \"maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz\" (UID: \"926b47a9-b1d0-4987-ad8c-9bacb98ff651\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz"
Jul 27 19:09:49.017200 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:49.017041    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/926b47a9-b1d0-4987-ad8c-9bacb98ff651-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz\" (UID: \"926b47a9-b1d0-4987-ad8c-9bacb98ff651\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz"
Jul 27 19:09:49.017626 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:49.017606    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/926b47a9-b1d0-4987-ad8c-9bacb98ff651-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz\" (UID: \"926b47a9-b1d0-4987-ad8c-9bacb98ff651\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz"
Jul 27 19:09:49.019675 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:49.019654    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/926b47a9-b1d0-4987-ad8c-9bacb98ff651-maas-api-tls\") pod \"maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz\" (UID: \"926b47a9-b1d0-4987-ad8c-9bacb98ff651\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz"
Jul 27 19:09:49.028782 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:49.028760    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hnsn\" (UniqueName: \"kubernetes.io/projected/926b47a9-b1d0-4987-ad8c-9bacb98ff651-kube-api-access-5hnsn\") pod \"maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz\" (UID: \"926b47a9-b1d0-4987-ad8c-9bacb98ff651\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz"
Jul 27 19:09:49.214466 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:49.214377    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz"
Jul 27 19:09:49.356892 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:49.356864    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz"]
Jul 27 19:09:49.358816 ip-10-0-128-196 kubenswrapper[2566]: W0727 19:09:49.358788    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926b47a9_b1d0_4987_ad8c_9bacb98ff651.slice/crio-3b35ae1e6aad2bf68801b0befe722c4acb94a7807f9e866d997a67723fdbac70 WatchSource:0}: Error finding container 3b35ae1e6aad2bf68801b0befe722c4acb94a7807f9e866d997a67723fdbac70: Status 404 returned error can't find the container with id 3b35ae1e6aad2bf68801b0befe722c4acb94a7807f9e866d997a67723fdbac70
Jul 27 19:09:49.360180 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:49.360161    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 19:09:49.992312 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:49.992278    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz" event={"ID":"926b47a9-b1d0-4987-ad8c-9bacb98ff651","Type":"ContainerStarted","Data":"e7785c6766745cc01f003b5d9913bb0f4b8a04018b8084febc711538bdf41fc9"}
Jul 27 19:09:49.992663 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:49.992328    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz" event={"ID":"926b47a9-b1d0-4987-ad8c-9bacb98ff651","Type":"ContainerStarted","Data":"3b35ae1e6aad2bf68801b0befe722c4acb94a7807f9e866d997a67723fdbac70"}
Jul 27 19:09:49.992663 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:49.992396    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz"
Jul 27 19:09:50.029265 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:50.029214    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz" podStartSLOduration=1.575224527 podStartE2EDuration="2.029198443s" totalImagesPullingTime="453.973916ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:09:50.027437952 +0000 UTC m=+2450.461695565" watchObservedRunningTime="2026-07-27 19:09:50.029198443 +0000 UTC m=+2450.463456136"
Jul 27 19:09:56.003019 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:56.002985    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz"
Jul 27 19:09:57.056274 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:57.056244    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p"]
Jul 27 19:09:57.060042 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:57.060018    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p"
Jul 27 19:09:57.062845 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:57.062818    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9abe8e-serving-cert\""
Jul 27 19:09:57.071906 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:57.071879    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p"]
Jul 27 19:09:57.088501 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:57.088463    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71a752a0-be84-4b3d-bf0f-862ceafc667c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p\" (UID: \"71a752a0-be84-4b3d-bf0f-862ceafc667c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p"
Jul 27 19:09:57.088636 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:57.088530    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71a752a0-be84-4b3d-bf0f-862ceafc667c-maas-api-tls\") pod \"maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p\" (UID: \"71a752a0-be84-4b3d-bf0f-862ceafc667c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p"
Jul 27 19:09:57.088636 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:57.088564    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tljts\" (UniqueName: \"kubernetes.io/projected/71a752a0-be84-4b3d-bf0f-862ceafc667c-kube-api-access-tljts\") pod \"maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p\" (UID: \"71a752a0-be84-4b3d-bf0f-862ceafc667c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p"
Jul 27 19:09:57.189476 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:57.189445    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71a752a0-be84-4b3d-bf0f-862ceafc667c-maas-api-tls\") pod \"maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p\" (UID: \"71a752a0-be84-4b3d-bf0f-862ceafc667c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p"
Jul 27 19:09:57.189625 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:57.189484    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tljts\" (UniqueName: \"kubernetes.io/projected/71a752a0-be84-4b3d-bf0f-862ceafc667c-kube-api-access-tljts\") pod \"maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p\" (UID: \"71a752a0-be84-4b3d-bf0f-862ceafc667c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p"
Jul 27 19:09:57.189625 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:57.189583    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71a752a0-be84-4b3d-bf0f-862ceafc667c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p\" (UID: \"71a752a0-be84-4b3d-bf0f-862ceafc667c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p"
Jul 27 19:09:57.189625 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:09:57.189601    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-serving-cert: secret "maas-api-e2e-inf-b-9abe8e-serving-cert" not found
Jul 27 19:09:57.189765 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:09:57.189653    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71a752a0-be84-4b3d-bf0f-862ceafc667c-maas-api-tls podName:71a752a0-be84-4b3d-bf0f-862ceafc667c nodeName:}" failed. No retries permitted until 2026-07-27 19:09:57.689637914 +0000 UTC m=+2458.123895503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/71a752a0-be84-4b3d-bf0f-862ceafc667c-maas-api-tls") pod "maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p" (UID: "71a752a0-be84-4b3d-bf0f-862ceafc667c") : secret "maas-api-e2e-inf-b-9abe8e-serving-cert" not found
Jul 27 19:09:57.190222 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:57.190206    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71a752a0-be84-4b3d-bf0f-862ceafc667c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p\" (UID: \"71a752a0-be84-4b3d-bf0f-862ceafc667c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p"
Jul 27 19:09:57.200062 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:57.200036    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tljts\" (UniqueName: \"kubernetes.io/projected/71a752a0-be84-4b3d-bf0f-862ceafc667c-kube-api-access-tljts\") pod \"maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p\" (UID: \"71a752a0-be84-4b3d-bf0f-862ceafc667c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p"
Jul 27 19:09:57.694701 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:57.694665    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71a752a0-be84-4b3d-bf0f-862ceafc667c-maas-api-tls\") pod \"maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p\" (UID: \"71a752a0-be84-4b3d-bf0f-862ceafc667c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p"
Jul 27 19:09:57.694878 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:09:57.694826    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-serving-cert: secret "maas-api-e2e-inf-b-9abe8e-serving-cert" not found
Jul 27 19:09:57.694923 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:09:57.694910    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71a752a0-be84-4b3d-bf0f-862ceafc667c-maas-api-tls podName:71a752a0-be84-4b3d-bf0f-862ceafc667c nodeName:}" failed. No retries permitted until 2026-07-27 19:09:58.694893019 +0000 UTC m=+2459.129150608 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/71a752a0-be84-4b3d-bf0f-862ceafc667c-maas-api-tls") pod "maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p" (UID: "71a752a0-be84-4b3d-bf0f-862ceafc667c") : secret "maas-api-e2e-inf-b-9abe8e-serving-cert" not found
Jul 27 19:09:58.705064 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:58.704967    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71a752a0-be84-4b3d-bf0f-862ceafc667c-maas-api-tls\") pod \"maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p\" (UID: \"71a752a0-be84-4b3d-bf0f-862ceafc667c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p"
Jul 27 19:09:58.707491 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:58.707465    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71a752a0-be84-4b3d-bf0f-862ceafc667c-maas-api-tls\") pod \"maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p\" (UID: \"71a752a0-be84-4b3d-bf0f-862ceafc667c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p"
Jul 27 19:09:58.872380 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:58.872344    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p"
Jul 27 19:09:59.009295 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:59.009269    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p"]
Jul 27 19:09:59.011545 ip-10-0-128-196 kubenswrapper[2566]: W0727 19:09:59.011512    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a752a0_be84_4b3d_bf0f_862ceafc667c.slice/crio-43a6d9efce11d379119a683a9f335c803b6e529a37ef9399c72403216542f214 WatchSource:0}: Error finding container 43a6d9efce11d379119a683a9f335c803b6e529a37ef9399c72403216542f214: Status 404 returned error can't find the container with id 43a6d9efce11d379119a683a9f335c803b6e529a37ef9399c72403216542f214
Jul 27 19:09:59.044517 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:09:59.044482    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p" event={"ID":"71a752a0-be84-4b3d-bf0f-862ceafc667c","Type":"ContainerStarted","Data":"43a6d9efce11d379119a683a9f335c803b6e529a37ef9399c72403216542f214"}
Jul 27 19:10:00.052540 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:00.052504    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p" event={"ID":"71a752a0-be84-4b3d-bf0f-862ceafc667c","Type":"ContainerStarted","Data":"0a696a4b272e73e0ddbe1fc0bae8dd9a07bf6f4ce408749099a580a6bd35e0c2"}
Jul 27 19:10:00.052961 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:00.052586    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p"
Jul 27 19:10:00.075977 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:00.075926    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p" podStartSLOduration=2.692832944 podStartE2EDuration="3.075909324s" totalImagesPullingTime="383.07638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:09:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:10:00.073893876 +0000 UTC m=+2460.508151488" watchObservedRunningTime="2026-07-27 19:10:00.075909324 +0000 UTC m=+2460.510166987"
Jul 27 19:10:06.062868 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:06.062836    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p"
Jul 27 19:10:17.593010 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:17.592971    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"]
Jul 27 19:10:17.596681 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:17.596662    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"
Jul 27 19:10:17.600311 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:17.600285    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f9caec\"/\"openshift-service-ca.crt\""
Jul 27 19:10:17.601893 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:17.601867    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f9caec\"/\"test-model-f9caec-kserve-self-signed-certs\""
Jul 27 19:10:17.602046 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:17.601973    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f9caec\"/\"default-dockercfg-rfvb4\""
Jul 27 19:10:17.602156 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:17.602135    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f9caec\"/\"kube-root-ca.crt\""
Jul 27 19:10:17.608856 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:17.608826    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"]
Jul 27 19:10:17.685297 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:17.685253    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad3611a-adb5-4c36-9571-24833b3dac95-dshm\") pod \"test-model-f9caec-kserve-6ff75fc555-4ndtf\" (UID: \"dad3611a-adb5-4c36-9571-24833b3dac95\") " pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"
Jul 27 19:10:17.685507 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:17.685320    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad3611a-adb5-4c36-9571-24833b3dac95-tls-certs\") pod \"test-model-f9caec-kserve-6ff75fc555-4ndtf\" (UID: \"dad3611a-adb5-4c36-9571-24833b3dac95\") " pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"
Jul 27 19:10:17.685507 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:17.685400    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2fdz\" (UniqueName: \"kubernetes.io/projected/dad3611a-adb5-4c36-9571-24833b3dac95-kube-api-access-k2fdz\") pod \"test-model-f9caec-kserve-6ff75fc555-4ndtf\" (UID: \"dad3611a-adb5-4c36-9571-24833b3dac95\") " pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"
Jul 27 19:10:17.685507 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:17.685450    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad3611a-adb5-4c36-9571-24833b3dac95-home\") pod \"test-model-f9caec-kserve-6ff75fc555-4ndtf\" (UID: \"dad3611a-adb5-4c36-9571-24833b3dac95\") " pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"
Jul 27 19:10:17.685507 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:17.685483    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad3611a-adb5-4c36-9571-24833b3dac95-model-cache\") pod \"test-model-f9caec-kserve-6ff75fc555-4ndtf\" (UID: \"dad3611a-adb5-4c36-9571-24833b3dac95\") " pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"
Jul 27 19:10:17.685729 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:17.685550    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad3611a-adb5-4c36-9571-24833b3dac95-tmp-dir\") pod \"test-model-f9caec-kserve-6ff75fc555-4ndtf\" (UID: \"dad3611a-adb5-4c36-9571-24833b3dac95\") " pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"
Jul 27 19:10:17.787054 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:17.787013    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad3611a-adb5-4c36-9571-24833b3dac95-tls-certs\") pod \"test-model-f9caec-kserve-6ff75fc555-4ndtf\" (UID: \"dad3611a-adb5-4c36-9571-24833b3dac95\") " pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"
Jul 27 19:10:17.787054 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:17.787056    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2fdz\" (UniqueName: \"kubernetes.io/projected/dad3611a-adb5-4c36-9571-24833b3dac95-kube-api-access-k2fdz\") pod \"test-model-f9caec-kserve-6ff75fc555-4ndtf\" (UID: \"dad3611a-adb5-4c36-9571-24833b3dac95\") " pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"
Jul 27 19:10:17.787331 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:17.787076    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad3611a-adb5-4c36-9571-24833b3dac95-home\") pod \"test-model-f9caec-kserve-6ff75fc555-4ndtf\" (UID: \"dad3611a-adb5-4c36-9571-24833b3dac95\") " pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"
Jul 27 19:10:17.787331 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:17.787092    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad3611a-adb5-4c36-9571-24833b3dac95-model-cache\") pod \"test-model-f9caec-kserve-6ff75fc555-4ndtf\" (UID: \"dad3611a-adb5-4c36-9571-24833b3dac95\") " pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"
Jul 27 19:10:17.787331 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:17.787169    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad3611a-adb5-4c36-9571-24833b3dac95-tmp-dir\") pod \"test-model-f9caec-kserve-6ff75fc555-4ndtf\" (UID: \"dad3611a-adb5-4c36-9571-24833b3dac95\") " pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"
Jul 27 19:10:17.787331 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:17.787218    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad3611a-adb5-4c36-9571-24833b3dac95-dshm\") pod \"test-model-f9caec-kserve-6ff75fc555-4ndtf\" (UID: \"dad3611a-adb5-4c36-9571-24833b3dac95\") " pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"
Jul 27 19:10:17.787612 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:17.787588    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad3611a-adb5-4c36-9571-24833b3dac95-home\") pod \"test-model-f9caec-kserve-6ff75fc555-4ndtf\" (UID: \"dad3611a-adb5-4c36-9571-24833b3dac95\") " pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"
Jul 27 19:10:17.787686 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:17.787626    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad3611a-adb5-4c36-9571-24833b3dac95-tmp-dir\") pod \"test-model-f9caec-kserve-6ff75fc555-4ndtf\" (UID: \"dad3611a-adb5-4c36-9571-24833b3dac95\") " pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"
Jul 27 19:10:17.787745 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:17.787717    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad3611a-adb5-4c36-9571-24833b3dac95-model-cache\") pod \"test-model-f9caec-kserve-6ff75fc555-4ndtf\" (UID: \"dad3611a-adb5-4c36-9571-24833b3dac95\") " pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"
Jul 27 19:10:17.789596 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:17.789577    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad3611a-adb5-4c36-9571-24833b3dac95-dshm\") pod \"test-model-f9caec-kserve-6ff75fc555-4ndtf\" (UID: \"dad3611a-adb5-4c36-9571-24833b3dac95\") " pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"
Jul 27 19:10:17.789851 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:17.789833    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad3611a-adb5-4c36-9571-24833b3dac95-tls-certs\") pod \"test-model-f9caec-kserve-6ff75fc555-4ndtf\" (UID: \"dad3611a-adb5-4c36-9571-24833b3dac95\") " pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"
Jul 27 19:10:17.796401 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:17.796371    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2fdz\" (UniqueName: \"kubernetes.io/projected/dad3611a-adb5-4c36-9571-24833b3dac95-kube-api-access-k2fdz\") pod \"test-model-f9caec-kserve-6ff75fc555-4ndtf\" (UID: \"dad3611a-adb5-4c36-9571-24833b3dac95\") " pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"
Jul 27 19:10:17.909711 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:17.909616    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"
Jul 27 19:10:18.057934 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:18.057906    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"]
Jul 27 19:10:18.059428 ip-10-0-128-196 kubenswrapper[2566]: W0727 19:10:18.059402    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddad3611a_adb5_4c36_9571_24833b3dac95.slice/crio-06cc6c3db62f7f192e012edd16f50a6f8585dd81ca2b5aac474bb95084631871 WatchSource:0}: Error finding container 06cc6c3db62f7f192e012edd16f50a6f8585dd81ca2b5aac474bb95084631871: Status 404 returned error can't find the container with id 06cc6c3db62f7f192e012edd16f50a6f8585dd81ca2b5aac474bb95084631871
Jul 27 19:10:18.158251 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:18.158213    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf" event={"ID":"dad3611a-adb5-4c36-9571-24833b3dac95","Type":"ContainerStarted","Data":"99a5fe63bc5986508fbb71b7ac6c48193046ec4fa4e4886324cd549e3245a441"}
Jul 27 19:10:18.158421 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:18.158266    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf" event={"ID":"dad3611a-adb5-4c36-9571-24833b3dac95","Type":"ContainerStarted","Data":"06cc6c3db62f7f192e012edd16f50a6f8585dd81ca2b5aac474bb95084631871"}
Jul 27 19:10:18.179580 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:18.179472    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf" podStartSLOduration=1.1794482020000001 podStartE2EDuration="1.179448202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:10:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:10:18.177411538 +0000 UTC m=+2478.611669394" watchObservedRunningTime="2026-07-27 19:10:18.179448202 +0000 UTC m=+2478.613705818"
Jul 27 19:10:27.909795 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:27.909757    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"
Jul 27 19:10:27.909795 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:27.909802    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"
Jul 27 19:10:27.922465 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:27.922439    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"
Jul 27 19:10:28.225218 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:28.225134    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"
Jul 27 19:10:34.896709 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:34.896674    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"]
Jul 27 19:10:34.900346 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:34.900323    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"
Jul 27 19:10:34.904811 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:34.904639    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9abe8e\"/\"openshift-service-ca.crt\""
Jul 27 19:10:34.906061 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:34.906039    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9abe8e\"/\"kube-root-ca.crt\""
Jul 27 19:10:34.906230 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:34.906095    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9abe8e\"/\"test-model-9abe8e-kserve-self-signed-certs\""
Jul 27 19:10:34.906230 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:34.906215    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9abe8e\"/\"default-dockercfg-gr7c9\""
Jul 27 19:10:34.914463 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:34.914438    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"]
Jul 27 19:10:34.937524 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:34.937486    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-dshm\") pod \"test-model-9abe8e-kserve-dd4595557-hcsc4\" (UID: \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\") " pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"
Jul 27 19:10:34.937524 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:34.937526    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-model-cache\") pod \"test-model-9abe8e-kserve-dd4595557-hcsc4\" (UID: \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\") " pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"
Jul 27 19:10:34.937737 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:34.937589    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-tls-certs\") pod \"test-model-9abe8e-kserve-dd4595557-hcsc4\" (UID: \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\") " pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"
Jul 27 19:10:34.937737 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:34.937626    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29nbb\" (UniqueName: \"kubernetes.io/projected/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-kube-api-access-29nbb\") pod \"test-model-9abe8e-kserve-dd4595557-hcsc4\" (UID: \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\") " pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"
Jul 27 19:10:34.937737 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:34.937664    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-tmp-dir\") pod \"test-model-9abe8e-kserve-dd4595557-hcsc4\" (UID: \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\") " pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"
Jul 27 19:10:34.937737 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:34.937689    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-home\") pod \"test-model-9abe8e-kserve-dd4595557-hcsc4\" (UID: \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\") " pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"
Jul 27 19:10:35.038519 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:35.038480    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-dshm\") pod \"test-model-9abe8e-kserve-dd4595557-hcsc4\" (UID: \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\") " pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"
Jul 27 19:10:35.038519 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:35.038516    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-model-cache\") pod \"test-model-9abe8e-kserve-dd4595557-hcsc4\" (UID: \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\") " pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"
Jul 27 19:10:35.038735 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:35.038541    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-tls-certs\") pod \"test-model-9abe8e-kserve-dd4595557-hcsc4\" (UID: \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\") " pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"
Jul 27 19:10:35.038735 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:35.038560    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29nbb\" (UniqueName: \"kubernetes.io/projected/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-kube-api-access-29nbb\") pod \"test-model-9abe8e-kserve-dd4595557-hcsc4\" (UID: \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\") " pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"
Jul 27 19:10:35.038735 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:35.038601    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-tmp-dir\") pod \"test-model-9abe8e-kserve-dd4595557-hcsc4\" (UID: \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\") " pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"
Jul 27 19:10:35.038735 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:35.038632    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-home\") pod \"test-model-9abe8e-kserve-dd4595557-hcsc4\" (UID: \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\") " pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"
Jul 27 19:10:35.038997 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:35.038975    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-model-cache\") pod \"test-model-9abe8e-kserve-dd4595557-hcsc4\" (UID: \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\") " pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"
Jul 27 19:10:35.039044 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:35.039031    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-tmp-dir\") pod \"test-model-9abe8e-kserve-dd4595557-hcsc4\" (UID: \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\") " pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"
Jul 27 19:10:35.039044 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:35.039037    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-home\") pod \"test-model-9abe8e-kserve-dd4595557-hcsc4\" (UID: \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\") " pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"
Jul 27 19:10:35.041036 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:35.041011    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-dshm\") pod \"test-model-9abe8e-kserve-dd4595557-hcsc4\" (UID: \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\") " pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"
Jul 27 19:10:35.041298 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:35.041282    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-tls-certs\") pod \"test-model-9abe8e-kserve-dd4595557-hcsc4\" (UID: \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\") " pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"
Jul 27 19:10:35.053878 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:35.053857    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29nbb\" (UniqueName: \"kubernetes.io/projected/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-kube-api-access-29nbb\") pod \"test-model-9abe8e-kserve-dd4595557-hcsc4\" (UID: \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\") " pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"
Jul 27 19:10:35.210888 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:35.210783    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"
Jul 27 19:10:35.353887 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:35.353862    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"]
Jul 27 19:10:35.356611 ip-10-0-128-196 kubenswrapper[2566]: W0727 19:10:35.356586    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a2e0dfa_cacf_4b88_b11f_52bc2eb91b9c.slice/crio-25b9d589155b0f3186440527170652552be1a0550a755134225472f938acbf39 WatchSource:0}: Error finding container 25b9d589155b0f3186440527170652552be1a0550a755134225472f938acbf39: Status 404 returned error can't find the container with id 25b9d589155b0f3186440527170652552be1a0550a755134225472f938acbf39
Jul 27 19:10:36.265306 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:36.265266    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4" event={"ID":"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c","Type":"ContainerStarted","Data":"d99afe276010093f7b4f07ae3aeee87856a2fbb94b432ef51586a24e429b5ad4"}
Jul 27 19:10:36.265306 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:36.265311    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4" event={"ID":"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c","Type":"ContainerStarted","Data":"25b9d589155b0f3186440527170652552be1a0550a755134225472f938acbf39"}
Jul 27 19:10:36.288002 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:36.287952    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4" podStartSLOduration=2.2879377659999998 podStartE2EDuration="2.287937766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:10:36.286164272 +0000 UTC m=+2496.720421897" watchObservedRunningTime="2026-07-27 19:10:36.287937766 +0000 UTC m=+2496.722195376"
Jul 27 19:10:45.211488 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:45.211450    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"
Jul 27 19:10:45.211877 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:45.211646    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"
Jul 27 19:10:45.224365 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:45.224331    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"
Jul 27 19:10:45.323960 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:10:45.323928    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"
Jul 27 19:11:16.375782 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:16.375746    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"]
Jul 27 19:11:17.444404 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:17.444366    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"]
Jul 27 19:11:27.963972 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:27.963938    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz"]
Jul 27 19:11:27.964467 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:27.964198    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz" podUID="926b47a9-b1d0-4987-ad8c-9bacb98ff651" containerName="maas-api" containerID="cri-o://e7785c6766745cc01f003b5d9913bb0f4b8a04018b8084febc711538bdf41fc9" gracePeriod=30
Jul 27 19:11:28.214821 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:28.214756    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz"
Jul 27 19:11:28.308238 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:28.308200    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/926b47a9-b1d0-4987-ad8c-9bacb98ff651-maas-api-tls\") pod \"926b47a9-b1d0-4987-ad8c-9bacb98ff651\" (UID: \"926b47a9-b1d0-4987-ad8c-9bacb98ff651\") "
Jul 27 19:11:28.308449 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:28.308255    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hnsn\" (UniqueName: \"kubernetes.io/projected/926b47a9-b1d0-4987-ad8c-9bacb98ff651-kube-api-access-5hnsn\") pod \"926b47a9-b1d0-4987-ad8c-9bacb98ff651\" (UID: \"926b47a9-b1d0-4987-ad8c-9bacb98ff651\") "
Jul 27 19:11:28.308449 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:28.308275    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/926b47a9-b1d0-4987-ad8c-9bacb98ff651-openshift-service-ca\") pod \"926b47a9-b1d0-4987-ad8c-9bacb98ff651\" (UID: \"926b47a9-b1d0-4987-ad8c-9bacb98ff651\") "
Jul 27 19:11:28.308695 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:28.308666    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/926b47a9-b1d0-4987-ad8c-9bacb98ff651-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "926b47a9-b1d0-4987-ad8c-9bacb98ff651" (UID: "926b47a9-b1d0-4987-ad8c-9bacb98ff651"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:11:28.310519 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:28.310494    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/926b47a9-b1d0-4987-ad8c-9bacb98ff651-kube-api-access-5hnsn" (OuterVolumeSpecName: "kube-api-access-5hnsn") pod "926b47a9-b1d0-4987-ad8c-9bacb98ff651" (UID: "926b47a9-b1d0-4987-ad8c-9bacb98ff651"). InnerVolumeSpecName "kube-api-access-5hnsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:11:28.310608 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:28.310566    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/926b47a9-b1d0-4987-ad8c-9bacb98ff651-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "926b47a9-b1d0-4987-ad8c-9bacb98ff651" (UID: "926b47a9-b1d0-4987-ad8c-9bacb98ff651"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:11:28.409546 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:28.409506    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/926b47a9-b1d0-4987-ad8c-9bacb98ff651-maas-api-tls\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:11:28.409718 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:28.409556    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hnsn\" (UniqueName: \"kubernetes.io/projected/926b47a9-b1d0-4987-ad8c-9bacb98ff651-kube-api-access-5hnsn\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:11:28.409718 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:28.409566    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/926b47a9-b1d0-4987-ad8c-9bacb98ff651-openshift-service-ca\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:11:28.555856 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:28.555823    2566 generic.go:356] "Generic (PLEG): container finished" podID="926b47a9-b1d0-4987-ad8c-9bacb98ff651" containerID="e7785c6766745cc01f003b5d9913bb0f4b8a04018b8084febc711538bdf41fc9" exitCode=0
Jul 27 19:11:28.556021 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:28.555867    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz"
Jul 27 19:11:28.556021 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:28.555908    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz" event={"ID":"926b47a9-b1d0-4987-ad8c-9bacb98ff651","Type":"ContainerDied","Data":"e7785c6766745cc01f003b5d9913bb0f4b8a04018b8084febc711538bdf41fc9"}
Jul 27 19:11:28.556021 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:28.555959    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz" event={"ID":"926b47a9-b1d0-4987-ad8c-9bacb98ff651","Type":"ContainerDied","Data":"3b35ae1e6aad2bf68801b0befe722c4acb94a7807f9e866d997a67723fdbac70"}
Jul 27 19:11:28.556021 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:28.555985    2566 scope.go:117] "RemoveContainer" containerID="e7785c6766745cc01f003b5d9913bb0f4b8a04018b8084febc711538bdf41fc9"
Jul 27 19:11:28.568472 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:28.568417    2566 scope.go:117] "RemoveContainer" containerID="e7785c6766745cc01f003b5d9913bb0f4b8a04018b8084febc711538bdf41fc9"
Jul 27 19:11:28.568926 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:11:28.568904    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7785c6766745cc01f003b5d9913bb0f4b8a04018b8084febc711538bdf41fc9\": container with ID starting with e7785c6766745cc01f003b5d9913bb0f4b8a04018b8084febc711538bdf41fc9 not found: ID does not exist" containerID="e7785c6766745cc01f003b5d9913bb0f4b8a04018b8084febc711538bdf41fc9"
Jul 27 19:11:28.569027 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:28.568934    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7785c6766745cc01f003b5d9913bb0f4b8a04018b8084febc711538bdf41fc9"} err="failed to get container status \"e7785c6766745cc01f003b5d9913bb0f4b8a04018b8084febc711538bdf41fc9\": rpc error: code = NotFound desc = could not find container \"e7785c6766745cc01f003b5d9913bb0f4b8a04018b8084febc711538bdf41fc9\": container with ID starting with e7785c6766745cc01f003b5d9913bb0f4b8a04018b8084febc711538bdf41fc9 not found: ID does not exist"
Jul 27 19:11:28.578478 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:28.578455    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz"]
Jul 27 19:11:28.581912 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:28.581891    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f9caec-66b45b77b9-zd8qz"]
Jul 27 19:11:30.124638 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:30.124602    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="926b47a9-b1d0-4987-ad8c-9bacb98ff651" path="/var/lib/kubelet/pods/926b47a9-b1d0-4987-ad8c-9bacb98ff651/volumes"
Jul 27 19:11:31.396098 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:31.396047    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf" podUID="dad3611a-adb5-4c36-9571-24833b3dac95" containerName="main" containerID="cri-o://99a5fe63bc5986508fbb71b7ac6c48193046ec4fa4e4886324cd549e3245a441" gracePeriod=45
Jul 27 19:11:31.575754 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:31.575722    2566 generic.go:356] "Generic (PLEG): container finished" podID="dad3611a-adb5-4c36-9571-24833b3dac95" containerID="99a5fe63bc5986508fbb71b7ac6c48193046ec4fa4e4886324cd549e3245a441" exitCode=0
Jul 27 19:11:31.575898 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:31.575793    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf" event={"ID":"dad3611a-adb5-4c36-9571-24833b3dac95","Type":"ContainerDied","Data":"99a5fe63bc5986508fbb71b7ac6c48193046ec4fa4e4886324cd549e3245a441"}
Jul 27 19:11:31.656436 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:31.656375    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"
Jul 27 19:11:31.736179 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:31.736148    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad3611a-adb5-4c36-9571-24833b3dac95-dshm\") pod \"dad3611a-adb5-4c36-9571-24833b3dac95\" (UID: \"dad3611a-adb5-4c36-9571-24833b3dac95\") "
Jul 27 19:11:31.736179 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:31.736182    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad3611a-adb5-4c36-9571-24833b3dac95-tls-certs\") pod \"dad3611a-adb5-4c36-9571-24833b3dac95\" (UID: \"dad3611a-adb5-4c36-9571-24833b3dac95\") "
Jul 27 19:11:31.736430 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:31.736209    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2fdz\" (UniqueName: \"kubernetes.io/projected/dad3611a-adb5-4c36-9571-24833b3dac95-kube-api-access-k2fdz\") pod \"dad3611a-adb5-4c36-9571-24833b3dac95\" (UID: \"dad3611a-adb5-4c36-9571-24833b3dac95\") "
Jul 27 19:11:31.736430 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:31.736235    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad3611a-adb5-4c36-9571-24833b3dac95-home\") pod \"dad3611a-adb5-4c36-9571-24833b3dac95\" (UID: \"dad3611a-adb5-4c36-9571-24833b3dac95\") "
Jul 27 19:11:31.736430 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:31.736250    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad3611a-adb5-4c36-9571-24833b3dac95-model-cache\") pod \"dad3611a-adb5-4c36-9571-24833b3dac95\" (UID: \"dad3611a-adb5-4c36-9571-24833b3dac95\") "
Jul 27 19:11:31.736430 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:31.736381    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad3611a-adb5-4c36-9571-24833b3dac95-tmp-dir\") pod \"dad3611a-adb5-4c36-9571-24833b3dac95\" (UID: \"dad3611a-adb5-4c36-9571-24833b3dac95\") "
Jul 27 19:11:31.736644 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:31.736512    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad3611a-adb5-4c36-9571-24833b3dac95-home" (OuterVolumeSpecName: "home") pod "dad3611a-adb5-4c36-9571-24833b3dac95" (UID: "dad3611a-adb5-4c36-9571-24833b3dac95"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:11:31.736697 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:31.736659    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad3611a-adb5-4c36-9571-24833b3dac95-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dad3611a-adb5-4c36-9571-24833b3dac95" (UID: "dad3611a-adb5-4c36-9571-24833b3dac95"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:11:31.736697 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:31.736688    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad3611a-adb5-4c36-9571-24833b3dac95-home\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:11:31.736806 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:31.736598    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad3611a-adb5-4c36-9571-24833b3dac95-model-cache" (OuterVolumeSpecName: "model-cache") pod "dad3611a-adb5-4c36-9571-24833b3dac95" (UID: "dad3611a-adb5-4c36-9571-24833b3dac95"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:11:31.738582 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:31.738555    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad3611a-adb5-4c36-9571-24833b3dac95-dshm" (OuterVolumeSpecName: "dshm") pod "dad3611a-adb5-4c36-9571-24833b3dac95" (UID: "dad3611a-adb5-4c36-9571-24833b3dac95"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:11:31.738582 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:31.738559    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad3611a-adb5-4c36-9571-24833b3dac95-kube-api-access-k2fdz" (OuterVolumeSpecName: "kube-api-access-k2fdz") pod "dad3611a-adb5-4c36-9571-24833b3dac95" (UID: "dad3611a-adb5-4c36-9571-24833b3dac95"). InnerVolumeSpecName "kube-api-access-k2fdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:11:31.738730 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:31.738598    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dad3611a-adb5-4c36-9571-24833b3dac95-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dad3611a-adb5-4c36-9571-24833b3dac95" (UID: "dad3611a-adb5-4c36-9571-24833b3dac95"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:11:31.837913 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:31.837876    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad3611a-adb5-4c36-9571-24833b3dac95-tmp-dir\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:11:31.837913 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:31.837903    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad3611a-adb5-4c36-9571-24833b3dac95-dshm\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:11:31.837913 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:31.837911    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad3611a-adb5-4c36-9571-24833b3dac95-tls-certs\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:11:31.837913 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:31.837919    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2fdz\" (UniqueName: \"kubernetes.io/projected/dad3611a-adb5-4c36-9571-24833b3dac95-kube-api-access-k2fdz\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:11:31.838203 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:31.837928    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad3611a-adb5-4c36-9571-24833b3dac95-model-cache\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:11:32.464022 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:32.463931    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4" podUID="9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c" containerName="main" containerID="cri-o://d99afe276010093f7b4f07ae3aeee87856a2fbb94b432ef51586a24e429b5ad4" gracePeriod=45
Jul 27 19:11:32.582334 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:32.582312    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"
Jul 27 19:11:32.582334 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:32.582316    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf" event={"ID":"dad3611a-adb5-4c36-9571-24833b3dac95","Type":"ContainerDied","Data":"06cc6c3db62f7f192e012edd16f50a6f8585dd81ca2b5aac474bb95084631871"}
Jul 27 19:11:32.582495 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:32.582374    2566 scope.go:117] "RemoveContainer" containerID="99a5fe63bc5986508fbb71b7ac6c48193046ec4fa4e4886324cd549e3245a441"
Jul 27 19:11:32.584509 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:32.584401    2566 generic.go:356] "Generic (PLEG): container finished" podID="9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c" containerID="d99afe276010093f7b4f07ae3aeee87856a2fbb94b432ef51586a24e429b5ad4" exitCode=0
Jul 27 19:11:32.584509 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:32.584455    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4" event={"ID":"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c","Type":"ContainerDied","Data":"d99afe276010093f7b4f07ae3aeee87856a2fbb94b432ef51586a24e429b5ad4"}
Jul 27 19:11:32.600554 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:32.600514    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"]
Jul 27 19:11:32.608013 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:32.607976    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f9caec/test-model-f9caec-kserve-6ff75fc555-4ndtf"]
Jul 27 19:11:32.726106 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:32.726036    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"
Jul 27 19:11:32.745880 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:32.745852    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29nbb\" (UniqueName: \"kubernetes.io/projected/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-kube-api-access-29nbb\") pod \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\" (UID: \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\") "
Jul 27 19:11:32.746046 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:32.745943    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-tls-certs\") pod \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\" (UID: \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\") "
Jul 27 19:11:32.746046 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:32.746034    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-dshm\") pod \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\" (UID: \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\") "
Jul 27 19:11:32.746201 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:32.746057    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-model-cache\") pod \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\" (UID: \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\") "
Jul 27 19:11:32.746201 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:32.746082    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-tmp-dir\") pod \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\" (UID: \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\") "
Jul 27 19:11:32.746201 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:32.746143    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-home\") pod \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\" (UID: \"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c\") "
Jul 27 19:11:32.746683 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:32.746531    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-model-cache" (OuterVolumeSpecName: "model-cache") pod "9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c" (UID: "9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:11:32.746683 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:32.746546    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c" (UID: "9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:11:32.746683 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:32.746642    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-home" (OuterVolumeSpecName: "home") pod "9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c" (UID: "9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:11:32.748900 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:32.748875    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-dshm" (OuterVolumeSpecName: "dshm") pod "9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c" (UID: "9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:11:32.749094 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:32.749065    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-kube-api-access-29nbb" (OuterVolumeSpecName: "kube-api-access-29nbb") pod "9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c" (UID: "9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c"). InnerVolumeSpecName "kube-api-access-29nbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:11:32.749466 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:32.749447    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c" (UID: "9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:11:32.847843 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:32.847805    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29nbb\" (UniqueName: \"kubernetes.io/projected/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-kube-api-access-29nbb\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:11:32.847843 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:32.847840    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-tls-certs\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:11:32.847843 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:32.847850    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-dshm\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:11:32.848086 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:32.847857    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-model-cache\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:11:32.848086 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:32.847865    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-tmp-dir\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:11:32.848086 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:32.847874    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c-home\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:11:33.592297 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:33.592262    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"
Jul 27 19:11:33.592732 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:33.592320    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4" event={"ID":"9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c","Type":"ContainerDied","Data":"25b9d589155b0f3186440527170652552be1a0550a755134225472f938acbf39"}
Jul 27 19:11:33.592732 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:33.592368    2566 scope.go:117] "RemoveContainer" containerID="d99afe276010093f7b4f07ae3aeee87856a2fbb94b432ef51586a24e429b5ad4"
Jul 27 19:11:33.614547 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:33.614524    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"]
Jul 27 19:11:33.618542 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:33.618520    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9abe8e/test-model-9abe8e-kserve-dd4595557-hcsc4"]
Jul 27 19:11:34.124370 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:34.124335    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c" path="/var/lib/kubelet/pods/9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c/volumes"
Jul 27 19:11:34.124732 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:34.124720    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dad3611a-adb5-4c36-9571-24833b3dac95" path="/var/lib/kubelet/pods/dad3611a-adb5-4c36-9571-24833b3dac95/volumes"
Jul 27 19:11:51.257980 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:51.257935    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p"]
Jul 27 19:11:51.258570 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:51.258279    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p" podUID="71a752a0-be84-4b3d-bf0f-862ceafc667c" containerName="maas-api" containerID="cri-o://0a696a4b272e73e0ddbe1fc0bae8dd9a07bf6f4ce408749099a580a6bd35e0c2" gracePeriod=30
Jul 27 19:11:51.520419 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:51.520350    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p"
Jul 27 19:11:51.613630 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:51.613600    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tljts\" (UniqueName: \"kubernetes.io/projected/71a752a0-be84-4b3d-bf0f-862ceafc667c-kube-api-access-tljts\") pod \"71a752a0-be84-4b3d-bf0f-862ceafc667c\" (UID: \"71a752a0-be84-4b3d-bf0f-862ceafc667c\") "
Jul 27 19:11:51.613810 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:51.613647    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71a752a0-be84-4b3d-bf0f-862ceafc667c-openshift-service-ca\") pod \"71a752a0-be84-4b3d-bf0f-862ceafc667c\" (UID: \"71a752a0-be84-4b3d-bf0f-862ceafc667c\") "
Jul 27 19:11:51.613810 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:51.613776    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71a752a0-be84-4b3d-bf0f-862ceafc667c-maas-api-tls\") pod \"71a752a0-be84-4b3d-bf0f-862ceafc667c\" (UID: \"71a752a0-be84-4b3d-bf0f-862ceafc667c\") "
Jul 27 19:11:51.614092 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:51.614067    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a752a0-be84-4b3d-bf0f-862ceafc667c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "71a752a0-be84-4b3d-bf0f-862ceafc667c" (UID: "71a752a0-be84-4b3d-bf0f-862ceafc667c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:11:51.615917 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:51.615882    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a752a0-be84-4b3d-bf0f-862ceafc667c-kube-api-access-tljts" (OuterVolumeSpecName: "kube-api-access-tljts") pod "71a752a0-be84-4b3d-bf0f-862ceafc667c" (UID: "71a752a0-be84-4b3d-bf0f-862ceafc667c"). InnerVolumeSpecName "kube-api-access-tljts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:11:51.616016 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:51.615881    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71a752a0-be84-4b3d-bf0f-862ceafc667c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "71a752a0-be84-4b3d-bf0f-862ceafc667c" (UID: "71a752a0-be84-4b3d-bf0f-862ceafc667c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:11:51.695936 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:51.695893    2566 generic.go:356] "Generic (PLEG): container finished" podID="71a752a0-be84-4b3d-bf0f-862ceafc667c" containerID="0a696a4b272e73e0ddbe1fc0bae8dd9a07bf6f4ce408749099a580a6bd35e0c2" exitCode=0
Jul 27 19:11:51.695936 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:51.695940    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p"
Jul 27 19:11:51.696218 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:51.695988    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p" event={"ID":"71a752a0-be84-4b3d-bf0f-862ceafc667c","Type":"ContainerDied","Data":"0a696a4b272e73e0ddbe1fc0bae8dd9a07bf6f4ce408749099a580a6bd35e0c2"}
Jul 27 19:11:51.696218 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:51.696039    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p" event={"ID":"71a752a0-be84-4b3d-bf0f-862ceafc667c","Type":"ContainerDied","Data":"43a6d9efce11d379119a683a9f335c803b6e529a37ef9399c72403216542f214"}
Jul 27 19:11:51.696218 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:51.696061    2566 scope.go:117] "RemoveContainer" containerID="0a696a4b272e73e0ddbe1fc0bae8dd9a07bf6f4ce408749099a580a6bd35e0c2"
Jul 27 19:11:51.705994 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:51.705972    2566 scope.go:117] "RemoveContainer" containerID="0a696a4b272e73e0ddbe1fc0bae8dd9a07bf6f4ce408749099a580a6bd35e0c2"
Jul 27 19:11:51.706320 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:11:51.706298    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a696a4b272e73e0ddbe1fc0bae8dd9a07bf6f4ce408749099a580a6bd35e0c2\": container with ID starting with 0a696a4b272e73e0ddbe1fc0bae8dd9a07bf6f4ce408749099a580a6bd35e0c2 not found: ID does not exist" containerID="0a696a4b272e73e0ddbe1fc0bae8dd9a07bf6f4ce408749099a580a6bd35e0c2"
Jul 27 19:11:51.706394 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:51.706328    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a696a4b272e73e0ddbe1fc0bae8dd9a07bf6f4ce408749099a580a6bd35e0c2"} err="failed to get container status \"0a696a4b272e73e0ddbe1fc0bae8dd9a07bf6f4ce408749099a580a6bd35e0c2\": rpc error: code = NotFound desc = could not find container \"0a696a4b272e73e0ddbe1fc0bae8dd9a07bf6f4ce408749099a580a6bd35e0c2\": container with ID starting with 0a696a4b272e73e0ddbe1fc0bae8dd9a07bf6f4ce408749099a580a6bd35e0c2 not found: ID does not exist"
Jul 27 19:11:51.714479 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:51.714455    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71a752a0-be84-4b3d-bf0f-862ceafc667c-maas-api-tls\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:11:51.714479 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:51.714478    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tljts\" (UniqueName: \"kubernetes.io/projected/71a752a0-be84-4b3d-bf0f-862ceafc667c-kube-api-access-tljts\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:11:51.714621 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:51.714488    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71a752a0-be84-4b3d-bf0f-862ceafc667c-openshift-service-ca\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:11:51.721095 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:51.721066    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p"]
Jul 27 19:11:51.726284 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:51.726258    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9abe8e-5697f744f9-bxm2p"]
Jul 27 19:11:52.126502 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:11:52.126460    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71a752a0-be84-4b3d-bf0f-862ceafc667c" path="/var/lib/kubelet/pods/71a752a0-be84-4b3d-bf0f-862ceafc667c/volumes"
Jul 27 19:12:06.621556 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:06.621515    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk"]
Jul 27 19:12:06.622099 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:06.622078    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c" containerName="main"
Jul 27 19:12:06.622234 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:06.622109    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c" containerName="main"
Jul 27 19:12:06.622234 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:06.622162    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="926b47a9-b1d0-4987-ad8c-9bacb98ff651" containerName="maas-api"
Jul 27 19:12:06.622234 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:06.622173    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="926b47a9-b1d0-4987-ad8c-9bacb98ff651" containerName="maas-api"
Jul 27 19:12:06.622234 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:06.622185    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71a752a0-be84-4b3d-bf0f-862ceafc667c" containerName="maas-api"
Jul 27 19:12:06.622234 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:06.622193    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="71a752a0-be84-4b3d-bf0f-862ceafc667c" containerName="maas-api"
Jul 27 19:12:06.622234 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:06.622217    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dad3611a-adb5-4c36-9571-24833b3dac95" containerName="main"
Jul 27 19:12:06.622234 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:06.622225    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dad3611a-adb5-4c36-9571-24833b3dac95" containerName="main"
Jul 27 19:12:06.622569 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:06.622319    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a2e0dfa-cacf-4b88-b11f-52bc2eb91b9c" containerName="main"
Jul 27 19:12:06.622569 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:06.622337    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="926b47a9-b1d0-4987-ad8c-9bacb98ff651" containerName="maas-api"
Jul 27 19:12:06.622569 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:06.622346    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dad3611a-adb5-4c36-9571-24833b3dac95" containerName="main"
Jul 27 19:12:06.622569 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:06.622358    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="71a752a0-be84-4b3d-bf0f-862ceafc667c" containerName="maas-api"
Jul 27 19:12:06.625320 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:06.625299    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk"
Jul 27 19:12:06.628279 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:06.628259    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-bf852f-serving-cert\""
Jul 27 19:12:06.628391 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:06.628352    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b9bn6\""
Jul 27 19:12:06.634721 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:06.634698    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk"]
Jul 27 19:12:06.756447 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:06.756416    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wws8t\" (UniqueName: \"kubernetes.io/projected/e9de8cbb-e6f0-4d12-a280-d2da592740ff-kube-api-access-wws8t\") pod \"maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk\" (UID: \"e9de8cbb-e6f0-4d12-a280-d2da592740ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk"
Jul 27 19:12:06.756647 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:06.756453    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9de8cbb-e6f0-4d12-a280-d2da592740ff-openshift-service-ca\") pod \"maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk\" (UID: \"e9de8cbb-e6f0-4d12-a280-d2da592740ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk"
Jul 27 19:12:06.756647 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:06.756537    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9de8cbb-e6f0-4d12-a280-d2da592740ff-maas-api-tls\") pod \"maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk\" (UID: \"e9de8cbb-e6f0-4d12-a280-d2da592740ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk"
Jul 27 19:12:06.857312 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:06.857283    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wws8t\" (UniqueName: \"kubernetes.io/projected/e9de8cbb-e6f0-4d12-a280-d2da592740ff-kube-api-access-wws8t\") pod \"maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk\" (UID: \"e9de8cbb-e6f0-4d12-a280-d2da592740ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk"
Jul 27 19:12:06.857312 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:06.857318    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9de8cbb-e6f0-4d12-a280-d2da592740ff-openshift-service-ca\") pod \"maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk\" (UID: \"e9de8cbb-e6f0-4d12-a280-d2da592740ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk"
Jul 27 19:12:06.857591 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:06.857369    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9de8cbb-e6f0-4d12-a280-d2da592740ff-maas-api-tls\") pod \"maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk\" (UID: \"e9de8cbb-e6f0-4d12-a280-d2da592740ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk"
Jul 27 19:12:06.857997 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:06.857974    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9de8cbb-e6f0-4d12-a280-d2da592740ff-openshift-service-ca\") pod \"maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk\" (UID: \"e9de8cbb-e6f0-4d12-a280-d2da592740ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk"
Jul 27 19:12:06.859926 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:06.859904    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9de8cbb-e6f0-4d12-a280-d2da592740ff-maas-api-tls\") pod \"maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk\" (UID: \"e9de8cbb-e6f0-4d12-a280-d2da592740ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk"
Jul 27 19:12:06.866718 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:06.866695    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wws8t\" (UniqueName: \"kubernetes.io/projected/e9de8cbb-e6f0-4d12-a280-d2da592740ff-kube-api-access-wws8t\") pod \"maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk\" (UID: \"e9de8cbb-e6f0-4d12-a280-d2da592740ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk"
Jul 27 19:12:06.936368 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:06.936274    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk"
Jul 27 19:12:07.085150 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:07.085108    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk"]
Jul 27 19:12:07.087203 ip-10-0-128-196 kubenswrapper[2566]: W0727 19:12:07.087176    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9de8cbb_e6f0_4d12_a280_d2da592740ff.slice/crio-d7c4e56655b9e561781327d1be1d62b9ee48ba0ce06b06d256b9975b883e5f44 WatchSource:0}: Error finding container d7c4e56655b9e561781327d1be1d62b9ee48ba0ce06b06d256b9975b883e5f44: Status 404 returned error can't find the container with id d7c4e56655b9e561781327d1be1d62b9ee48ba0ce06b06d256b9975b883e5f44
Jul 27 19:12:07.787982 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:07.787940    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk" event={"ID":"e9de8cbb-e6f0-4d12-a280-d2da592740ff","Type":"ContainerStarted","Data":"7f57c5e8622aaede1a50625e2c568ebae0d54aa3bd263101227b22d32f73c49c"}
Jul 27 19:12:07.788380 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:07.788038    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk" event={"ID":"e9de8cbb-e6f0-4d12-a280-d2da592740ff","Type":"ContainerStarted","Data":"d7c4e56655b9e561781327d1be1d62b9ee48ba0ce06b06d256b9975b883e5f44"}
Jul 27 19:12:07.788380 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:07.788070    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk"
Jul 27 19:12:07.824670 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:07.824613    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk" podStartSLOduration=1.240979199 podStartE2EDuration="1.824595453s" totalImagesPullingTime="583.616254ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:12:07.822475061 +0000 UTC m=+2588.256732673" watchObservedRunningTime="2026-07-27 19:12:07.824595453 +0000 UTC m=+2588.258853115"
Jul 27 19:12:13.798812 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:13.798783    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk"
Jul 27 19:12:14.874970 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:14.873079    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d"]
Jul 27 19:12:14.881585 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:14.881552    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d"
Jul 27 19:12:14.886078 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:14.886054    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e1ba46-serving-cert\""
Jul 27 19:12:14.918070 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:14.918034    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d"]
Jul 27 19:12:15.035796 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:15.035741    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkczq\" (UniqueName: \"kubernetes.io/projected/85fbd9f0-37a4-4767-9597-a697a6e935b2-kube-api-access-zkczq\") pod \"maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d\" (UID: \"85fbd9f0-37a4-4767-9597-a697a6e935b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d"
Jul 27 19:12:15.036006 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:15.035833    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85fbd9f0-37a4-4767-9597-a697a6e935b2-openshift-service-ca\") pod \"maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d\" (UID: \"85fbd9f0-37a4-4767-9597-a697a6e935b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d"
Jul 27 19:12:15.036006 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:15.035865    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85fbd9f0-37a4-4767-9597-a697a6e935b2-maas-api-tls\") pod \"maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d\" (UID: \"85fbd9f0-37a4-4767-9597-a697a6e935b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d"
Jul 27 19:12:15.136531 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:15.136427    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkczq\" (UniqueName: \"kubernetes.io/projected/85fbd9f0-37a4-4767-9597-a697a6e935b2-kube-api-access-zkczq\") pod \"maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d\" (UID: \"85fbd9f0-37a4-4767-9597-a697a6e935b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d"
Jul 27 19:12:15.136531 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:15.136502    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85fbd9f0-37a4-4767-9597-a697a6e935b2-openshift-service-ca\") pod \"maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d\" (UID: \"85fbd9f0-37a4-4767-9597-a697a6e935b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d"
Jul 27 19:12:15.136777 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:15.136538    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85fbd9f0-37a4-4767-9597-a697a6e935b2-maas-api-tls\") pod \"maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d\" (UID: \"85fbd9f0-37a4-4767-9597-a697a6e935b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d"
Jul 27 19:12:15.136777 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:12:15.136680    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-serving-cert: secret "maas-api-e2e-api-b-e1ba46-serving-cert" not found
Jul 27 19:12:15.136777 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:12:15.136760    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85fbd9f0-37a4-4767-9597-a697a6e935b2-maas-api-tls podName:85fbd9f0-37a4-4767-9597-a697a6e935b2 nodeName:}" failed. No retries permitted until 2026-07-27 19:12:15.636738858 +0000 UTC m=+2596.070996448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/85fbd9f0-37a4-4767-9597-a697a6e935b2-maas-api-tls") pod "maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d" (UID: "85fbd9f0-37a4-4767-9597-a697a6e935b2") : secret "maas-api-e2e-api-b-e1ba46-serving-cert" not found
Jul 27 19:12:15.137362 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:15.137338    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85fbd9f0-37a4-4767-9597-a697a6e935b2-openshift-service-ca\") pod \"maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d\" (UID: \"85fbd9f0-37a4-4767-9597-a697a6e935b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d"
Jul 27 19:12:15.159209 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:15.159170    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkczq\" (UniqueName: \"kubernetes.io/projected/85fbd9f0-37a4-4767-9597-a697a6e935b2-kube-api-access-zkczq\") pod \"maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d\" (UID: \"85fbd9f0-37a4-4767-9597-a697a6e935b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d"
Jul 27 19:12:15.639834 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:15.639794    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85fbd9f0-37a4-4767-9597-a697a6e935b2-maas-api-tls\") pod \"maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d\" (UID: \"85fbd9f0-37a4-4767-9597-a697a6e935b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d"
Jul 27 19:12:15.642457 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:15.642435    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85fbd9f0-37a4-4767-9597-a697a6e935b2-maas-api-tls\") pod \"maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d\" (UID: \"85fbd9f0-37a4-4767-9597-a697a6e935b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d"
Jul 27 19:12:15.792048 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:15.792008    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d"
Jul 27 19:12:15.962007 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:15.961981    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d"]
Jul 27 19:12:15.963806 ip-10-0-128-196 kubenswrapper[2566]: W0727 19:12:15.963778    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85fbd9f0_37a4_4767_9597_a697a6e935b2.slice/crio-2ec3bd2f19d13a710c12b7c73fee39a12cc6b2cb30dbd0f3c8a1c3755d9080e5 WatchSource:0}: Error finding container 2ec3bd2f19d13a710c12b7c73fee39a12cc6b2cb30dbd0f3c8a1c3755d9080e5: Status 404 returned error can't find the container with id 2ec3bd2f19d13a710c12b7c73fee39a12cc6b2cb30dbd0f3c8a1c3755d9080e5
Jul 27 19:12:16.843950 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:16.843913    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d" event={"ID":"85fbd9f0-37a4-4767-9597-a697a6e935b2","Type":"ContainerStarted","Data":"30e62fafb46a80e2ad526b6d8710535ebd2c0d4a0a036cf72b5713f3909f572d"}
Jul 27 19:12:16.844172 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:16.843971    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d" event={"ID":"85fbd9f0-37a4-4767-9597-a697a6e935b2","Type":"ContainerStarted","Data":"2ec3bd2f19d13a710c12b7c73fee39a12cc6b2cb30dbd0f3c8a1c3755d9080e5"}
Jul 27 19:12:16.844172 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:16.843992    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d"
Jul 27 19:12:16.878943 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:16.878880    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d" podStartSLOduration=2.335862353 podStartE2EDuration="2.878861249s" totalImagesPullingTime="542.998896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:12:16.876517952 +0000 UTC m=+2597.310775564" watchObservedRunningTime="2026-07-27 19:12:16.878861249 +0000 UTC m=+2597.313118859"
Jul 27 19:12:22.855934 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:22.855855    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d"
Jul 27 19:12:42.769786 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:42.769749    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk"]
Jul 27 19:12:42.770273 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:42.770000    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk" podUID="e9de8cbb-e6f0-4d12-a280-d2da592740ff" containerName="maas-api" containerID="cri-o://7f57c5e8622aaede1a50625e2c568ebae0d54aa3bd263101227b22d32f73c49c" gracePeriod=30
Jul 27 19:12:42.996434 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:42.996282    2566 generic.go:356] "Generic (PLEG): container finished" podID="e9de8cbb-e6f0-4d12-a280-d2da592740ff" containerID="7f57c5e8622aaede1a50625e2c568ebae0d54aa3bd263101227b22d32f73c49c" exitCode=0
Jul 27 19:12:42.996434 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:42.996369    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk" event={"ID":"e9de8cbb-e6f0-4d12-a280-d2da592740ff","Type":"ContainerDied","Data":"7f57c5e8622aaede1a50625e2c568ebae0d54aa3bd263101227b22d32f73c49c"}
Jul 27 19:12:43.040110 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:43.040082    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk"
Jul 27 19:12:43.197089 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:43.197057    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wws8t\" (UniqueName: \"kubernetes.io/projected/e9de8cbb-e6f0-4d12-a280-d2da592740ff-kube-api-access-wws8t\") pod \"e9de8cbb-e6f0-4d12-a280-d2da592740ff\" (UID: \"e9de8cbb-e6f0-4d12-a280-d2da592740ff\") "
Jul 27 19:12:43.197287 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:43.197098    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9de8cbb-e6f0-4d12-a280-d2da592740ff-openshift-service-ca\") pod \"e9de8cbb-e6f0-4d12-a280-d2da592740ff\" (UID: \"e9de8cbb-e6f0-4d12-a280-d2da592740ff\") "
Jul 27 19:12:43.197287 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:43.197130    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9de8cbb-e6f0-4d12-a280-d2da592740ff-maas-api-tls\") pod \"e9de8cbb-e6f0-4d12-a280-d2da592740ff\" (UID: \"e9de8cbb-e6f0-4d12-a280-d2da592740ff\") "
Jul 27 19:12:43.197495 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:43.197471    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9de8cbb-e6f0-4d12-a280-d2da592740ff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e9de8cbb-e6f0-4d12-a280-d2da592740ff" (UID: "e9de8cbb-e6f0-4d12-a280-d2da592740ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:12:43.199356 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:43.199335    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9de8cbb-e6f0-4d12-a280-d2da592740ff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e9de8cbb-e6f0-4d12-a280-d2da592740ff" (UID: "e9de8cbb-e6f0-4d12-a280-d2da592740ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:12:43.199424 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:43.199408    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9de8cbb-e6f0-4d12-a280-d2da592740ff-kube-api-access-wws8t" (OuterVolumeSpecName: "kube-api-access-wws8t") pod "e9de8cbb-e6f0-4d12-a280-d2da592740ff" (UID: "e9de8cbb-e6f0-4d12-a280-d2da592740ff"). InnerVolumeSpecName "kube-api-access-wws8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:12:43.298166 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:43.298044    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wws8t\" (UniqueName: \"kubernetes.io/projected/e9de8cbb-e6f0-4d12-a280-d2da592740ff-kube-api-access-wws8t\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:12:43.298166 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:43.298075    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9de8cbb-e6f0-4d12-a280-d2da592740ff-openshift-service-ca\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:12:43.298166 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:43.298085    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9de8cbb-e6f0-4d12-a280-d2da592740ff-maas-api-tls\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:12:44.010678 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:44.010649    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk"
Jul 27 19:12:44.011091 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:44.010683    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk" event={"ID":"e9de8cbb-e6f0-4d12-a280-d2da592740ff","Type":"ContainerDied","Data":"d7c4e56655b9e561781327d1be1d62b9ee48ba0ce06b06d256b9975b883e5f44"}
Jul 27 19:12:44.011091 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:44.010768    2566 scope.go:117] "RemoveContainer" containerID="7f57c5e8622aaede1a50625e2c568ebae0d54aa3bd263101227b22d32f73c49c"
Jul 27 19:12:44.036275 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:44.036246    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk"]
Jul 27 19:12:44.042738 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:44.042710    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf852f-558c7f9968-pnvhk"]
Jul 27 19:12:44.123741 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:12:44.123700    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9de8cbb-e6f0-4d12-a280-d2da592740ff" path="/var/lib/kubelet/pods/e9de8cbb-e6f0-4d12-a280-d2da592740ff/volumes"
Jul 27 19:13:11.288436 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:11.288405    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d"]
Jul 27 19:13:11.288840 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:11.288600    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d" podUID="85fbd9f0-37a4-4767-9597-a697a6e935b2" containerName="maas-api" containerID="cri-o://30e62fafb46a80e2ad526b6d8710535ebd2c0d4a0a036cf72b5713f3909f572d" gracePeriod=30
Jul 27 19:13:11.538398 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:11.538374    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d"
Jul 27 19:13:11.655372 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:11.655275    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85fbd9f0-37a4-4767-9597-a697a6e935b2-openshift-service-ca\") pod \"85fbd9f0-37a4-4767-9597-a697a6e935b2\" (UID: \"85fbd9f0-37a4-4767-9597-a697a6e935b2\") "
Jul 27 19:13:11.655372 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:11.655329    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85fbd9f0-37a4-4767-9597-a697a6e935b2-maas-api-tls\") pod \"85fbd9f0-37a4-4767-9597-a697a6e935b2\" (UID: \"85fbd9f0-37a4-4767-9597-a697a6e935b2\") "
Jul 27 19:13:11.655372 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:11.655365    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkczq\" (UniqueName: \"kubernetes.io/projected/85fbd9f0-37a4-4767-9597-a697a6e935b2-kube-api-access-zkczq\") pod \"85fbd9f0-37a4-4767-9597-a697a6e935b2\" (UID: \"85fbd9f0-37a4-4767-9597-a697a6e935b2\") "
Jul 27 19:13:11.655778 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:11.655749    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85fbd9f0-37a4-4767-9597-a697a6e935b2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "85fbd9f0-37a4-4767-9597-a697a6e935b2" (UID: "85fbd9f0-37a4-4767-9597-a697a6e935b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:13:11.657647 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:11.657615    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85fbd9f0-37a4-4767-9597-a697a6e935b2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "85fbd9f0-37a4-4767-9597-a697a6e935b2" (UID: "85fbd9f0-37a4-4767-9597-a697a6e935b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:13:11.657647 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:11.657634    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85fbd9f0-37a4-4767-9597-a697a6e935b2-kube-api-access-zkczq" (OuterVolumeSpecName: "kube-api-access-zkczq") pod "85fbd9f0-37a4-4767-9597-a697a6e935b2" (UID: "85fbd9f0-37a4-4767-9597-a697a6e935b2"). InnerVolumeSpecName "kube-api-access-zkczq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:13:11.756140 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:11.756084    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkczq\" (UniqueName: \"kubernetes.io/projected/85fbd9f0-37a4-4767-9597-a697a6e935b2-kube-api-access-zkczq\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:13:11.756140 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:11.756113    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85fbd9f0-37a4-4767-9597-a697a6e935b2-openshift-service-ca\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:13:11.756140 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:11.756143    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85fbd9f0-37a4-4767-9597-a697a6e935b2-maas-api-tls\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:13:12.167876 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:12.167834    2566 generic.go:356] "Generic (PLEG): container finished" podID="85fbd9f0-37a4-4767-9597-a697a6e935b2" containerID="30e62fafb46a80e2ad526b6d8710535ebd2c0d4a0a036cf72b5713f3909f572d" exitCode=0
Jul 27 19:13:12.168055 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:12.167895    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d"
Jul 27 19:13:12.168055 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:12.167916    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d" event={"ID":"85fbd9f0-37a4-4767-9597-a697a6e935b2","Type":"ContainerDied","Data":"30e62fafb46a80e2ad526b6d8710535ebd2c0d4a0a036cf72b5713f3909f572d"}
Jul 27 19:13:12.168055 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:12.167967    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d" event={"ID":"85fbd9f0-37a4-4767-9597-a697a6e935b2","Type":"ContainerDied","Data":"2ec3bd2f19d13a710c12b7c73fee39a12cc6b2cb30dbd0f3c8a1c3755d9080e5"}
Jul 27 19:13:12.168055 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:12.167989    2566 scope.go:117] "RemoveContainer" containerID="30e62fafb46a80e2ad526b6d8710535ebd2c0d4a0a036cf72b5713f3909f572d"
Jul 27 19:13:12.177079 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:12.177061    2566 scope.go:117] "RemoveContainer" containerID="30e62fafb46a80e2ad526b6d8710535ebd2c0d4a0a036cf72b5713f3909f572d"
Jul 27 19:13:12.177370 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:13:12.177350    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30e62fafb46a80e2ad526b6d8710535ebd2c0d4a0a036cf72b5713f3909f572d\": container with ID starting with 30e62fafb46a80e2ad526b6d8710535ebd2c0d4a0a036cf72b5713f3909f572d not found: ID does not exist" containerID="30e62fafb46a80e2ad526b6d8710535ebd2c0d4a0a036cf72b5713f3909f572d"
Jul 27 19:13:12.177435 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:12.177380    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30e62fafb46a80e2ad526b6d8710535ebd2c0d4a0a036cf72b5713f3909f572d"} err="failed to get container status \"30e62fafb46a80e2ad526b6d8710535ebd2c0d4a0a036cf72b5713f3909f572d\": rpc error: code = NotFound desc = could not find container \"30e62fafb46a80e2ad526b6d8710535ebd2c0d4a0a036cf72b5713f3909f572d\": container with ID starting with 30e62fafb46a80e2ad526b6d8710535ebd2c0d4a0a036cf72b5713f3909f572d not found: ID does not exist"
Jul 27 19:13:12.191887 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:12.191862    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d"]
Jul 27 19:13:12.199999 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:12.199977    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e1ba46-56f554b98c-wxd6d"]
Jul 27 19:13:14.124277 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:14.124245    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85fbd9f0-37a4-4767-9597-a697a6e935b2" path="/var/lib/kubelet/pods/85fbd9f0-37a4-4767-9597-a697a6e935b2/volumes"
Jul 27 19:13:27.357379 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:27.357346    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9"]
Jul 27 19:13:27.357769 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:27.357756    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9de8cbb-e6f0-4d12-a280-d2da592740ff" containerName="maas-api"
Jul 27 19:13:27.357813 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:27.357771    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9de8cbb-e6f0-4d12-a280-d2da592740ff" containerName="maas-api"
Jul 27 19:13:27.357813 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:27.357780    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85fbd9f0-37a4-4767-9597-a697a6e935b2" containerName="maas-api"
Jul 27 19:13:27.357813 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:27.357785    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="85fbd9f0-37a4-4767-9597-a697a6e935b2" containerName="maas-api"
Jul 27 19:13:27.357914 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:27.357852    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9de8cbb-e6f0-4d12-a280-d2da592740ff" containerName="maas-api"
Jul 27 19:13:27.357914 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:27.357862    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="85fbd9f0-37a4-4767-9597-a697a6e935b2" containerName="maas-api"
Jul 27 19:13:27.362260 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:27.362243    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9"
Jul 27 19:13:27.366589 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:27.366571    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-7f7473-serving-cert\""
Jul 27 19:13:27.367191 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:27.367171    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b9bn6\""
Jul 27 19:13:27.392959 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:27.392936    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9"]
Jul 27 19:13:27.513553 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:27.513521    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srwtg\" (UniqueName: \"kubernetes.io/projected/6987ede9-ad85-48df-b047-5eaf2e65b1cc-kube-api-access-srwtg\") pod \"maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9\" (UID: \"6987ede9-ad85-48df-b047-5eaf2e65b1cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9"
Jul 27 19:13:27.513553 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:27.513559    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6987ede9-ad85-48df-b047-5eaf2e65b1cc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9\" (UID: \"6987ede9-ad85-48df-b047-5eaf2e65b1cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9"
Jul 27 19:13:27.513781 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:27.513657    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6987ede9-ad85-48df-b047-5eaf2e65b1cc-maas-api-tls\") pod \"maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9\" (UID: \"6987ede9-ad85-48df-b047-5eaf2e65b1cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9"
Jul 27 19:13:27.614427 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:27.614333    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6987ede9-ad85-48df-b047-5eaf2e65b1cc-maas-api-tls\") pod \"maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9\" (UID: \"6987ede9-ad85-48df-b047-5eaf2e65b1cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9"
Jul 27 19:13:27.614427 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:27.614400    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srwtg\" (UniqueName: \"kubernetes.io/projected/6987ede9-ad85-48df-b047-5eaf2e65b1cc-kube-api-access-srwtg\") pod \"maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9\" (UID: \"6987ede9-ad85-48df-b047-5eaf2e65b1cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9"
Jul 27 19:13:27.614427 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:27.614421    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6987ede9-ad85-48df-b047-5eaf2e65b1cc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9\" (UID: \"6987ede9-ad85-48df-b047-5eaf2e65b1cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9"
Jul 27 19:13:27.614653 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:13:27.614500    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-serving-cert: secret "maas-api-e2e-shared-a-7f7473-serving-cert" not found
Jul 27 19:13:27.614653 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:13:27.614592    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6987ede9-ad85-48df-b047-5eaf2e65b1cc-maas-api-tls podName:6987ede9-ad85-48df-b047-5eaf2e65b1cc nodeName:}" failed. No retries permitted until 2026-07-27 19:13:28.114570286 +0000 UTC m=+2668.548827874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6987ede9-ad85-48df-b047-5eaf2e65b1cc-maas-api-tls") pod "maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9" (UID: "6987ede9-ad85-48df-b047-5eaf2e65b1cc") : secret "maas-api-e2e-shared-a-7f7473-serving-cert" not found
Jul 27 19:13:27.614974 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:27.614955    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6987ede9-ad85-48df-b047-5eaf2e65b1cc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9\" (UID: \"6987ede9-ad85-48df-b047-5eaf2e65b1cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9"
Jul 27 19:13:27.628703 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:27.628682    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srwtg\" (UniqueName: \"kubernetes.io/projected/6987ede9-ad85-48df-b047-5eaf2e65b1cc-kube-api-access-srwtg\") pod \"maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9\" (UID: \"6987ede9-ad85-48df-b047-5eaf2e65b1cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9"
Jul 27 19:13:28.120190 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:28.120155    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6987ede9-ad85-48df-b047-5eaf2e65b1cc-maas-api-tls\") pod \"maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9\" (UID: \"6987ede9-ad85-48df-b047-5eaf2e65b1cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9"
Jul 27 19:13:28.125005 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:28.124976    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6987ede9-ad85-48df-b047-5eaf2e65b1cc-maas-api-tls\") pod \"maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9\" (UID: \"6987ede9-ad85-48df-b047-5eaf2e65b1cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9"
Jul 27 19:13:28.272304 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:28.272273    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9"
Jul 27 19:13:28.433447 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:28.433422    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9"]
Jul 27 19:13:28.435020 ip-10-0-128-196 kubenswrapper[2566]: W0727 19:13:28.434989    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6987ede9_ad85_48df_b047_5eaf2e65b1cc.slice/crio-5b266f9bcb30527f6b9593f4f8c24e88de9d62023bdcf4600619278bfb0322e6 WatchSource:0}: Error finding container 5b266f9bcb30527f6b9593f4f8c24e88de9d62023bdcf4600619278bfb0322e6: Status 404 returned error can't find the container with id 5b266f9bcb30527f6b9593f4f8c24e88de9d62023bdcf4600619278bfb0322e6
Jul 27 19:13:29.258600 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:29.258510    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9" event={"ID":"6987ede9-ad85-48df-b047-5eaf2e65b1cc","Type":"ContainerStarted","Data":"3ec2d370f5cd742d09b70e567efbaeb604359a87a8566fb5c3c127c01f1d9163"}
Jul 27 19:13:29.258600 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:29.258560    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9" event={"ID":"6987ede9-ad85-48df-b047-5eaf2e65b1cc","Type":"ContainerStarted","Data":"5b266f9bcb30527f6b9593f4f8c24e88de9d62023bdcf4600619278bfb0322e6"}
Jul 27 19:13:29.258926 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:29.258641    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9"
Jul 27 19:13:29.290095 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:29.290038    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9" podStartSLOduration=1.847443071 podStartE2EDuration="2.290021658s" totalImagesPullingTime="442.578587ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:13:29.289422844 +0000 UTC m=+2669.723680457" watchObservedRunningTime="2026-07-27 19:13:29.290021658 +0000 UTC m=+2669.724279268"
Jul 27 19:13:35.269795 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:35.269767    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9"
Jul 27 19:13:35.447328 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:35.447295    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt"]
Jul 27 19:13:35.451100 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:35.451080    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt"
Jul 27 19:13:35.458014 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:35.457998    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-6710a9-serving-cert\""
Jul 27 19:13:35.508203 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:35.508175    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt"]
Jul 27 19:13:35.591135 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:35.591096    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z4hb\" (UniqueName: \"kubernetes.io/projected/14a2e734-99a5-4d2f-9f23-8f8676ba2111-kube-api-access-2z4hb\") pod \"maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt\" (UID: \"14a2e734-99a5-4d2f-9f23-8f8676ba2111\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt"
Jul 27 19:13:35.591296 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:35.591213    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14a2e734-99a5-4d2f-9f23-8f8676ba2111-openshift-service-ca\") pod \"maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt\" (UID: \"14a2e734-99a5-4d2f-9f23-8f8676ba2111\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt"
Jul 27 19:13:35.591296 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:35.591266    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14a2e734-99a5-4d2f-9f23-8f8676ba2111-maas-api-tls\") pod \"maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt\" (UID: \"14a2e734-99a5-4d2f-9f23-8f8676ba2111\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt"
Jul 27 19:13:35.692132 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:35.692088    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z4hb\" (UniqueName: \"kubernetes.io/projected/14a2e734-99a5-4d2f-9f23-8f8676ba2111-kube-api-access-2z4hb\") pod \"maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt\" (UID: \"14a2e734-99a5-4d2f-9f23-8f8676ba2111\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt"
Jul 27 19:13:35.692315 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:35.692172    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14a2e734-99a5-4d2f-9f23-8f8676ba2111-openshift-service-ca\") pod \"maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt\" (UID: \"14a2e734-99a5-4d2f-9f23-8f8676ba2111\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt"
Jul 27 19:13:35.692315 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:35.692205    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14a2e734-99a5-4d2f-9f23-8f8676ba2111-maas-api-tls\") pod \"maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt\" (UID: \"14a2e734-99a5-4d2f-9f23-8f8676ba2111\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt"
Jul 27 19:13:35.692879 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:35.692853    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14a2e734-99a5-4d2f-9f23-8f8676ba2111-openshift-service-ca\") pod \"maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt\" (UID: \"14a2e734-99a5-4d2f-9f23-8f8676ba2111\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt"
Jul 27 19:13:35.694657 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:35.694632    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14a2e734-99a5-4d2f-9f23-8f8676ba2111-maas-api-tls\") pod \"maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt\" (UID: \"14a2e734-99a5-4d2f-9f23-8f8676ba2111\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt"
Jul 27 19:13:35.738445 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:35.738420    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z4hb\" (UniqueName: \"kubernetes.io/projected/14a2e734-99a5-4d2f-9f23-8f8676ba2111-kube-api-access-2z4hb\") pod \"maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt\" (UID: \"14a2e734-99a5-4d2f-9f23-8f8676ba2111\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt"
Jul 27 19:13:35.760874 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:35.760846    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt"
Jul 27 19:13:35.922998 ip-10-0-128-196 kubenswrapper[2566]: W0727 19:13:35.922973    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a2e734_99a5_4d2f_9f23_8f8676ba2111.slice/crio-3d0ab9e821606be588c0fc87aa64d237b9e45fa7942b7e4105f873b22a6653d4 WatchSource:0}: Error finding container 3d0ab9e821606be588c0fc87aa64d237b9e45fa7942b7e4105f873b22a6653d4: Status 404 returned error can't find the container with id 3d0ab9e821606be588c0fc87aa64d237b9e45fa7942b7e4105f873b22a6653d4
Jul 27 19:13:35.930316 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:35.930295    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt"]
Jul 27 19:13:36.297465 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:36.297429    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt" event={"ID":"14a2e734-99a5-4d2f-9f23-8f8676ba2111","Type":"ContainerStarted","Data":"3d0ab9e821606be588c0fc87aa64d237b9e45fa7942b7e4105f873b22a6653d4"}
Jul 27 19:13:37.304505 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:37.304465    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt" event={"ID":"14a2e734-99a5-4d2f-9f23-8f8676ba2111","Type":"ContainerStarted","Data":"e264cf049bff9ffbc7a521abe826749b0f543ab70faf5a7ecb4c15af8e35d2c1"}
Jul 27 19:13:37.304952 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:37.304613    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt"
Jul 27 19:13:37.373947 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:37.373887    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt" podStartSLOduration=1.8022457570000001 podStartE2EDuration="2.373862687s" totalImagesPullingTime="571.61693ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:13:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:13:37.373307809 +0000 UTC m=+2677.807565421" watchObservedRunningTime="2026-07-27 19:13:37.373862687 +0000 UTC m=+2677.808120299"
Jul 27 19:13:43.315136 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:43.315093    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt"
Jul 27 19:13:51.205947 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:51.205907    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"]
Jul 27 19:13:51.209575 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:51.209554    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"
Jul 27 19:13:51.209979 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:51.209954    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq7wj\" (UniqueName: \"kubernetes.io/projected/03f4c088-d3d6-48da-98af-deea6cba8145-kube-api-access-cq7wj\") pod \"auth-test-model-7f7473-kserve-846fbfd9f-knjbd\" (UID: \"03f4c088-d3d6-48da-98af-deea6cba8145\") " pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"
Jul 27 19:13:51.210066 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:51.209996    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03f4c088-d3d6-48da-98af-deea6cba8145-tmp-dir\") pod \"auth-test-model-7f7473-kserve-846fbfd9f-knjbd\" (UID: \"03f4c088-d3d6-48da-98af-deea6cba8145\") " pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"
Jul 27 19:13:51.210066 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:51.210024    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03f4c088-d3d6-48da-98af-deea6cba8145-dshm\") pod \"auth-test-model-7f7473-kserve-846fbfd9f-knjbd\" (UID: \"03f4c088-d3d6-48da-98af-deea6cba8145\") " pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"
Jul 27 19:13:51.210066 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:51.210050    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03f4c088-d3d6-48da-98af-deea6cba8145-tls-certs\") pod \"auth-test-model-7f7473-kserve-846fbfd9f-knjbd\" (UID: \"03f4c088-d3d6-48da-98af-deea6cba8145\") " pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"
Jul 27 19:13:51.210269 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:51.210165    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03f4c088-d3d6-48da-98af-deea6cba8145-home\") pod \"auth-test-model-7f7473-kserve-846fbfd9f-knjbd\" (UID: \"03f4c088-d3d6-48da-98af-deea6cba8145\") " pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"
Jul 27 19:13:51.210269 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:51.210211    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03f4c088-d3d6-48da-98af-deea6cba8145-model-cache\") pod \"auth-test-model-7f7473-kserve-846fbfd9f-knjbd\" (UID: \"03f4c088-d3d6-48da-98af-deea6cba8145\") " pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"
Jul 27 19:13:51.218685 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:51.218670    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7f7473\"/\"kube-root-ca.crt\""
Jul 27 19:13:51.220157 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:51.220140    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7f7473\"/\"default-dockercfg-5ncxf\""
Jul 27 19:13:51.220214 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:51.220147    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7f7473\"/\"auth-test-model-7f7473-kserve-self-signed-certs\""
Jul 27 19:13:51.220214 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:51.220147    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7f7473\"/\"openshift-service-ca.crt\""
Jul 27 19:13:51.238957 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:51.238929    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"]
Jul 27 19:13:51.310857 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:51.310822    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03f4c088-d3d6-48da-98af-deea6cba8145-tls-certs\") pod \"auth-test-model-7f7473-kserve-846fbfd9f-knjbd\" (UID: \"03f4c088-d3d6-48da-98af-deea6cba8145\") " pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"
Jul 27 19:13:51.311000 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:51.310864    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03f4c088-d3d6-48da-98af-deea6cba8145-home\") pod \"auth-test-model-7f7473-kserve-846fbfd9f-knjbd\" (UID: \"03f4c088-d3d6-48da-98af-deea6cba8145\") " pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"
Jul 27 19:13:51.311000 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:51.310910    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03f4c088-d3d6-48da-98af-deea6cba8145-model-cache\") pod \"auth-test-model-7f7473-kserve-846fbfd9f-knjbd\" (UID: \"03f4c088-d3d6-48da-98af-deea6cba8145\") " pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"
Jul 27 19:13:51.311000 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:51.310994    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq7wj\" (UniqueName: \"kubernetes.io/projected/03f4c088-d3d6-48da-98af-deea6cba8145-kube-api-access-cq7wj\") pod \"auth-test-model-7f7473-kserve-846fbfd9f-knjbd\" (UID: \"03f4c088-d3d6-48da-98af-deea6cba8145\") " pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"
Jul 27 19:13:51.311191 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:51.311021    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03f4c088-d3d6-48da-98af-deea6cba8145-tmp-dir\") pod \"auth-test-model-7f7473-kserve-846fbfd9f-knjbd\" (UID: \"03f4c088-d3d6-48da-98af-deea6cba8145\") " pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"
Jul 27 19:13:51.311191 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:51.311044    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03f4c088-d3d6-48da-98af-deea6cba8145-dshm\") pod \"auth-test-model-7f7473-kserve-846fbfd9f-knjbd\" (UID: \"03f4c088-d3d6-48da-98af-deea6cba8145\") " pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"
Jul 27 19:13:51.311392 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:51.311370    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03f4c088-d3d6-48da-98af-deea6cba8145-home\") pod \"auth-test-model-7f7473-kserve-846fbfd9f-knjbd\" (UID: \"03f4c088-d3d6-48da-98af-deea6cba8145\") " pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"
Jul 27 19:13:51.311458 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:51.311400    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03f4c088-d3d6-48da-98af-deea6cba8145-model-cache\") pod \"auth-test-model-7f7473-kserve-846fbfd9f-knjbd\" (UID: \"03f4c088-d3d6-48da-98af-deea6cba8145\") " pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"
Jul 27 19:13:51.311458 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:51.311437    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03f4c088-d3d6-48da-98af-deea6cba8145-tmp-dir\") pod \"auth-test-model-7f7473-kserve-846fbfd9f-knjbd\" (UID: \"03f4c088-d3d6-48da-98af-deea6cba8145\") " pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"
Jul 27 19:13:51.313481 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:51.313459    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03f4c088-d3d6-48da-98af-deea6cba8145-dshm\") pod \"auth-test-model-7f7473-kserve-846fbfd9f-knjbd\" (UID: \"03f4c088-d3d6-48da-98af-deea6cba8145\") " pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"
Jul 27 19:13:51.313675 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:51.313657    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03f4c088-d3d6-48da-98af-deea6cba8145-tls-certs\") pod \"auth-test-model-7f7473-kserve-846fbfd9f-knjbd\" (UID: \"03f4c088-d3d6-48da-98af-deea6cba8145\") " pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"
Jul 27 19:13:51.325798 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:51.325769    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq7wj\" (UniqueName: \"kubernetes.io/projected/03f4c088-d3d6-48da-98af-deea6cba8145-kube-api-access-cq7wj\") pod \"auth-test-model-7f7473-kserve-846fbfd9f-knjbd\" (UID: \"03f4c088-d3d6-48da-98af-deea6cba8145\") " pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"
Jul 27 19:13:51.519503 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:51.519422    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"
Jul 27 19:13:51.679433 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:51.679410    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"]
Jul 27 19:13:51.681079 ip-10-0-128-196 kubenswrapper[2566]: W0727 19:13:51.681052    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03f4c088_d3d6_48da_98af_deea6cba8145.slice/crio-2c22617df91de7f16251f1ac03f9ecb0fe7062b9a823a7d8fa126ff7ac2d9ccc WatchSource:0}: Error finding container 2c22617df91de7f16251f1ac03f9ecb0fe7062b9a823a7d8fa126ff7ac2d9ccc: Status 404 returned error can't find the container with id 2c22617df91de7f16251f1ac03f9ecb0fe7062b9a823a7d8fa126ff7ac2d9ccc
Jul 27 19:13:52.386067 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:52.386021    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd" event={"ID":"03f4c088-d3d6-48da-98af-deea6cba8145","Type":"ContainerStarted","Data":"8a195f0fcedac1d007d3c1156be21bdc7f7db8d25d2c3a5398a6c7c200653c13"}
Jul 27 19:13:52.386465 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:52.386073    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd" event={"ID":"03f4c088-d3d6-48da-98af-deea6cba8145","Type":"ContainerStarted","Data":"2c22617df91de7f16251f1ac03f9ecb0fe7062b9a823a7d8fa126ff7ac2d9ccc"}
Jul 27 19:13:52.417476 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:13:52.417429    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd" podStartSLOduration=1.417417527 podStartE2EDuration="1.417417527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:13:52.416248926 +0000 UTC m=+2692.850506547" watchObservedRunningTime="2026-07-27 19:13:52.417417527 +0000 UTC m=+2692.851675137"
Jul 27 19:14:00.287963 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:00.287936    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 19:14:00.300997 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:00.300976    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 19:14:01.520426 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:01.520385    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"
Jul 27 19:14:01.520807 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:01.520451    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"
Jul 27 19:14:01.532944 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:01.532917    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"
Jul 27 19:14:02.450521 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:02.450488    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"
Jul 27 19:14:33.428588 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:33.428550    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"]
Jul 27 19:14:33.432257 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:33.432240    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"
Jul 27 19:14:33.437573 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:33.437546    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6710a9\"/\"default-dockercfg-r8f6c\""
Jul 27 19:14:33.437573 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:33.437570    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6710a9\"/\"kube-root-ca.crt\""
Jul 27 19:14:33.437780 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:33.437552    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6710a9\"/\"openshift-service-ca.crt\""
Jul 27 19:14:33.437780 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:33.437548    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6710a9\"/\"auth-test-model-6710a9-kserve-self-signed-certs\""
Jul 27 19:14:33.448965 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:33.448943    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"]
Jul 27 19:14:33.582309 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:33.582274    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32403748-8961-4b62-81c8-81f9945d4f28-model-cache\") pod \"auth-test-model-6710a9-kserve-7bcf577699-4z7rf\" (UID: \"32403748-8961-4b62-81c8-81f9945d4f28\") " pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"
Jul 27 19:14:33.582471 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:33.582323    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32403748-8961-4b62-81c8-81f9945d4f28-dshm\") pod \"auth-test-model-6710a9-kserve-7bcf577699-4z7rf\" (UID: \"32403748-8961-4b62-81c8-81f9945d4f28\") " pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"
Jul 27 19:14:33.582471 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:33.582405    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32403748-8961-4b62-81c8-81f9945d4f28-tls-certs\") pod \"auth-test-model-6710a9-kserve-7bcf577699-4z7rf\" (UID: \"32403748-8961-4b62-81c8-81f9945d4f28\") " pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"
Jul 27 19:14:33.582471 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:33.582454    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsqfw\" (UniqueName: \"kubernetes.io/projected/32403748-8961-4b62-81c8-81f9945d4f28-kube-api-access-qsqfw\") pod \"auth-test-model-6710a9-kserve-7bcf577699-4z7rf\" (UID: \"32403748-8961-4b62-81c8-81f9945d4f28\") " pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"
Jul 27 19:14:33.582668 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:33.582488    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32403748-8961-4b62-81c8-81f9945d4f28-home\") pod \"auth-test-model-6710a9-kserve-7bcf577699-4z7rf\" (UID: \"32403748-8961-4b62-81c8-81f9945d4f28\") " pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"
Jul 27 19:14:33.582668 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:33.582572    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32403748-8961-4b62-81c8-81f9945d4f28-tmp-dir\") pod \"auth-test-model-6710a9-kserve-7bcf577699-4z7rf\" (UID: \"32403748-8961-4b62-81c8-81f9945d4f28\") " pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"
Jul 27 19:14:33.683340 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:33.683242    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32403748-8961-4b62-81c8-81f9945d4f28-tls-certs\") pod \"auth-test-model-6710a9-kserve-7bcf577699-4z7rf\" (UID: \"32403748-8961-4b62-81c8-81f9945d4f28\") " pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"
Jul 27 19:14:33.683340 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:33.683290    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsqfw\" (UniqueName: \"kubernetes.io/projected/32403748-8961-4b62-81c8-81f9945d4f28-kube-api-access-qsqfw\") pod \"auth-test-model-6710a9-kserve-7bcf577699-4z7rf\" (UID: \"32403748-8961-4b62-81c8-81f9945d4f28\") " pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"
Jul 27 19:14:33.683340 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:33.683313    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32403748-8961-4b62-81c8-81f9945d4f28-home\") pod \"auth-test-model-6710a9-kserve-7bcf577699-4z7rf\" (UID: \"32403748-8961-4b62-81c8-81f9945d4f28\") " pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"
Jul 27 19:14:33.683632 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:33.683362    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32403748-8961-4b62-81c8-81f9945d4f28-tmp-dir\") pod \"auth-test-model-6710a9-kserve-7bcf577699-4z7rf\" (UID: \"32403748-8961-4b62-81c8-81f9945d4f28\") " pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"
Jul 27 19:14:33.683632 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:33.683393    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32403748-8961-4b62-81c8-81f9945d4f28-model-cache\") pod \"auth-test-model-6710a9-kserve-7bcf577699-4z7rf\" (UID: \"32403748-8961-4b62-81c8-81f9945d4f28\") " pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"
Jul 27 19:14:33.683632 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:33.683426    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32403748-8961-4b62-81c8-81f9945d4f28-dshm\") pod \"auth-test-model-6710a9-kserve-7bcf577699-4z7rf\" (UID: \"32403748-8961-4b62-81c8-81f9945d4f28\") " pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"
Jul 27 19:14:33.683864 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:33.683822    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32403748-8961-4b62-81c8-81f9945d4f28-home\") pod \"auth-test-model-6710a9-kserve-7bcf577699-4z7rf\" (UID: \"32403748-8961-4b62-81c8-81f9945d4f28\") " pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"
Jul 27 19:14:33.683864 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:33.683826    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32403748-8961-4b62-81c8-81f9945d4f28-model-cache\") pod \"auth-test-model-6710a9-kserve-7bcf577699-4z7rf\" (UID: \"32403748-8961-4b62-81c8-81f9945d4f28\") " pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"
Jul 27 19:14:33.683947 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:33.683884    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32403748-8961-4b62-81c8-81f9945d4f28-tmp-dir\") pod \"auth-test-model-6710a9-kserve-7bcf577699-4z7rf\" (UID: \"32403748-8961-4b62-81c8-81f9945d4f28\") " pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"
Jul 27 19:14:33.685712 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:33.685690    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32403748-8961-4b62-81c8-81f9945d4f28-dshm\") pod \"auth-test-model-6710a9-kserve-7bcf577699-4z7rf\" (UID: \"32403748-8961-4b62-81c8-81f9945d4f28\") " pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"
Jul 27 19:14:33.685842 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:33.685821    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32403748-8961-4b62-81c8-81f9945d4f28-tls-certs\") pod \"auth-test-model-6710a9-kserve-7bcf577699-4z7rf\" (UID: \"32403748-8961-4b62-81c8-81f9945d4f28\") " pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"
Jul 27 19:14:33.694057 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:33.694036    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsqfw\" (UniqueName: \"kubernetes.io/projected/32403748-8961-4b62-81c8-81f9945d4f28-kube-api-access-qsqfw\") pod \"auth-test-model-6710a9-kserve-7bcf577699-4z7rf\" (UID: \"32403748-8961-4b62-81c8-81f9945d4f28\") " pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"
Jul 27 19:14:33.742616 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:33.742586    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"
Jul 27 19:14:33.895046 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:33.895021    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"]
Jul 27 19:14:33.896393 ip-10-0-128-196 kubenswrapper[2566]: W0727 19:14:33.896366    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32403748_8961_4b62_81c8_81f9945d4f28.slice/crio-c2b80832865e5f98b41440dbb8df4ddba61d49da8b5343862d3718d1bc2635cb WatchSource:0}: Error finding container c2b80832865e5f98b41440dbb8df4ddba61d49da8b5343862d3718d1bc2635cb: Status 404 returned error can't find the container with id c2b80832865e5f98b41440dbb8df4ddba61d49da8b5343862d3718d1bc2635cb
Jul 27 19:14:34.620457 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:34.620420    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf" event={"ID":"32403748-8961-4b62-81c8-81f9945d4f28","Type":"ContainerStarted","Data":"7c15b097604890dcc58324a287971bb3998f12babe805056daecb49d8cf05180"}
Jul 27 19:14:34.620457 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:34.620461    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf" event={"ID":"32403748-8961-4b62-81c8-81f9945d4f28","Type":"ContainerStarted","Data":"c2b80832865e5f98b41440dbb8df4ddba61d49da8b5343862d3718d1bc2635cb"}
Jul 27 19:14:43.743390 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:43.743356    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"
Jul 27 19:14:43.743802 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:43.743408    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"
Jul 27 19:14:43.755780 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:43.755753    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"
Jul 27 19:14:43.776930 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:43.776882    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf" podStartSLOduration=10.776868264 podStartE2EDuration="10.776868264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:14:34.668328302 +0000 UTC m=+2735.102585913" watchObservedRunningTime="2026-07-27 19:14:43.776868264 +0000 UTC m=+2744.211125896"
Jul 27 19:14:44.685510 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:14:44.685476    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"
Jul 27 19:18:59.080926 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:18:59.080891    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"]
Jul 27 19:19:00.057038 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:00.056992    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"]
Jul 27 19:19:00.323688 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:00.323611    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 19:19:00.340015 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:00.339986    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 19:19:14.100968 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:14.100925    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd" podUID="03f4c088-d3d6-48da-98af-deea6cba8145" containerName="main" containerID="cri-o://8a195f0fcedac1d007d3c1156be21bdc7f7db8d25d2c3a5398a6c7c200653c13" gracePeriod=45
Jul 27 19:19:14.358948 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:14.358875    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"
Jul 27 19:19:14.377394 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:14.377367    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cq7wj\" (UniqueName: \"kubernetes.io/projected/03f4c088-d3d6-48da-98af-deea6cba8145-kube-api-access-cq7wj\") pod \"03f4c088-d3d6-48da-98af-deea6cba8145\" (UID: \"03f4c088-d3d6-48da-98af-deea6cba8145\") "
Jul 27 19:19:14.377569 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:14.377415    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03f4c088-d3d6-48da-98af-deea6cba8145-dshm\") pod \"03f4c088-d3d6-48da-98af-deea6cba8145\" (UID: \"03f4c088-d3d6-48da-98af-deea6cba8145\") "
Jul 27 19:19:14.377569 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:14.377442    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03f4c088-d3d6-48da-98af-deea6cba8145-tls-certs\") pod \"03f4c088-d3d6-48da-98af-deea6cba8145\" (UID: \"03f4c088-d3d6-48da-98af-deea6cba8145\") "
Jul 27 19:19:14.377569 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:14.377474    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03f4c088-d3d6-48da-98af-deea6cba8145-model-cache\") pod \"03f4c088-d3d6-48da-98af-deea6cba8145\" (UID: \"03f4c088-d3d6-48da-98af-deea6cba8145\") "
Jul 27 19:19:14.377569 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:14.377515    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03f4c088-d3d6-48da-98af-deea6cba8145-home\") pod \"03f4c088-d3d6-48da-98af-deea6cba8145\" (UID: \"03f4c088-d3d6-48da-98af-deea6cba8145\") "
Jul 27 19:19:14.377569 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:14.377553    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03f4c088-d3d6-48da-98af-deea6cba8145-tmp-dir\") pod \"03f4c088-d3d6-48da-98af-deea6cba8145\" (UID: \"03f4c088-d3d6-48da-98af-deea6cba8145\") "
Jul 27 19:19:14.377825 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:14.377773    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f4c088-d3d6-48da-98af-deea6cba8145-model-cache" (OuterVolumeSpecName: "model-cache") pod "03f4c088-d3d6-48da-98af-deea6cba8145" (UID: "03f4c088-d3d6-48da-98af-deea6cba8145"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:19:14.377825 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:14.377784    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f4c088-d3d6-48da-98af-deea6cba8145-home" (OuterVolumeSpecName: "home") pod "03f4c088-d3d6-48da-98af-deea6cba8145" (UID: "03f4c088-d3d6-48da-98af-deea6cba8145"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:19:14.377825 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:14.377813    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f4c088-d3d6-48da-98af-deea6cba8145-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "03f4c088-d3d6-48da-98af-deea6cba8145" (UID: "03f4c088-d3d6-48da-98af-deea6cba8145"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:19:14.377992 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:14.377970    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03f4c088-d3d6-48da-98af-deea6cba8145-model-cache\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:19:14.377992 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:14.377993    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03f4c088-d3d6-48da-98af-deea6cba8145-home\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:19:14.378393 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:14.378003    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03f4c088-d3d6-48da-98af-deea6cba8145-tmp-dir\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:19:14.379866 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:14.379846    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03f4c088-d3d6-48da-98af-deea6cba8145-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "03f4c088-d3d6-48da-98af-deea6cba8145" (UID: "03f4c088-d3d6-48da-98af-deea6cba8145"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:19:14.379957 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:14.379904    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f4c088-d3d6-48da-98af-deea6cba8145-kube-api-access-cq7wj" (OuterVolumeSpecName: "kube-api-access-cq7wj") pod "03f4c088-d3d6-48da-98af-deea6cba8145" (UID: "03f4c088-d3d6-48da-98af-deea6cba8145"). InnerVolumeSpecName "kube-api-access-cq7wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:19:14.379957 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:14.379908    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f4c088-d3d6-48da-98af-deea6cba8145-dshm" (OuterVolumeSpecName: "dshm") pod "03f4c088-d3d6-48da-98af-deea6cba8145" (UID: "03f4c088-d3d6-48da-98af-deea6cba8145"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:19:14.478950 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:14.478913    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq7wj\" (UniqueName: \"kubernetes.io/projected/03f4c088-d3d6-48da-98af-deea6cba8145-kube-api-access-cq7wj\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:19:14.478950 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:14.478941    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03f4c088-d3d6-48da-98af-deea6cba8145-dshm\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:19:14.478950 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:14.478955    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03f4c088-d3d6-48da-98af-deea6cba8145-tls-certs\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:19:15.078159 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.078066    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf" podUID="32403748-8961-4b62-81c8-81f9945d4f28" containerName="main" containerID="cri-o://7c15b097604890dcc58324a287971bb3998f12babe805056daecb49d8cf05180" gracePeriod=45
Jul 27 19:19:15.159650 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.159619    2566 generic.go:356] "Generic (PLEG): container finished" podID="03f4c088-d3d6-48da-98af-deea6cba8145" containerID="8a195f0fcedac1d007d3c1156be21bdc7f7db8d25d2c3a5398a6c7c200653c13" exitCode=0
Jul 27 19:19:15.160007 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.159662    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"
Jul 27 19:19:15.160007 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.159691    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd" event={"ID":"03f4c088-d3d6-48da-98af-deea6cba8145","Type":"ContainerDied","Data":"8a195f0fcedac1d007d3c1156be21bdc7f7db8d25d2c3a5398a6c7c200653c13"}
Jul 27 19:19:15.160007 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.159722    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd" event={"ID":"03f4c088-d3d6-48da-98af-deea6cba8145","Type":"ContainerDied","Data":"2c22617df91de7f16251f1ac03f9ecb0fe7062b9a823a7d8fa126ff7ac2d9ccc"}
Jul 27 19:19:15.160007 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.159739    2566 scope.go:117] "RemoveContainer" containerID="8a195f0fcedac1d007d3c1156be21bdc7f7db8d25d2c3a5398a6c7c200653c13"
Jul 27 19:19:15.184477 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.184448    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"]
Jul 27 19:19:15.190737 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.190716    2566 scope.go:117] "RemoveContainer" containerID="8a195f0fcedac1d007d3c1156be21bdc7f7db8d25d2c3a5398a6c7c200653c13"
Jul 27 19:19:15.191072 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:19:15.191048    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a195f0fcedac1d007d3c1156be21bdc7f7db8d25d2c3a5398a6c7c200653c13\": container with ID starting with 8a195f0fcedac1d007d3c1156be21bdc7f7db8d25d2c3a5398a6c7c200653c13 not found: ID does not exist" containerID="8a195f0fcedac1d007d3c1156be21bdc7f7db8d25d2c3a5398a6c7c200653c13"
Jul 27 19:19:15.191183 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.191080    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a195f0fcedac1d007d3c1156be21bdc7f7db8d25d2c3a5398a6c7c200653c13"} err="failed to get container status \"8a195f0fcedac1d007d3c1156be21bdc7f7db8d25d2c3a5398a6c7c200653c13\": rpc error: code = NotFound desc = could not find container \"8a195f0fcedac1d007d3c1156be21bdc7f7db8d25d2c3a5398a6c7c200653c13\": container with ID starting with 8a195f0fcedac1d007d3c1156be21bdc7f7db8d25d2c3a5398a6c7c200653c13 not found: ID does not exist"
Jul 27 19:19:15.191874 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.191851    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-7f7473/auth-test-model-7f7473-kserve-846fbfd9f-knjbd"]
Jul 27 19:19:15.336537 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.336474    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"
Jul 27 19:19:15.386551 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.386517    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32403748-8961-4b62-81c8-81f9945d4f28-home\") pod \"32403748-8961-4b62-81c8-81f9945d4f28\" (UID: \"32403748-8961-4b62-81c8-81f9945d4f28\") "
Jul 27 19:19:15.386723 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.386588    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsqfw\" (UniqueName: \"kubernetes.io/projected/32403748-8961-4b62-81c8-81f9945d4f28-kube-api-access-qsqfw\") pod \"32403748-8961-4b62-81c8-81f9945d4f28\" (UID: \"32403748-8961-4b62-81c8-81f9945d4f28\") "
Jul 27 19:19:15.386723 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.386644    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32403748-8961-4b62-81c8-81f9945d4f28-tmp-dir\") pod \"32403748-8961-4b62-81c8-81f9945d4f28\" (UID: \"32403748-8961-4b62-81c8-81f9945d4f28\") "
Jul 27 19:19:15.386723 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.386697    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32403748-8961-4b62-81c8-81f9945d4f28-model-cache\") pod \"32403748-8961-4b62-81c8-81f9945d4f28\" (UID: \"32403748-8961-4b62-81c8-81f9945d4f28\") "
Jul 27 19:19:15.386905 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.386758    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32403748-8961-4b62-81c8-81f9945d4f28-dshm\") pod \"32403748-8961-4b62-81c8-81f9945d4f28\" (UID: \"32403748-8961-4b62-81c8-81f9945d4f28\") "
Jul 27 19:19:15.386905 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.386780    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32403748-8961-4b62-81c8-81f9945d4f28-tls-certs\") pod \"32403748-8961-4b62-81c8-81f9945d4f28\" (UID: \"32403748-8961-4b62-81c8-81f9945d4f28\") "
Jul 27 19:19:15.386905 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.386821    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32403748-8961-4b62-81c8-81f9945d4f28-home" (OuterVolumeSpecName: "home") pod "32403748-8961-4b62-81c8-81f9945d4f28" (UID: "32403748-8961-4b62-81c8-81f9945d4f28"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:19:15.387054 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.386980    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32403748-8961-4b62-81c8-81f9945d4f28-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "32403748-8961-4b62-81c8-81f9945d4f28" (UID: "32403748-8961-4b62-81c8-81f9945d4f28"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:19:15.387054 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.386993    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32403748-8961-4b62-81c8-81f9945d4f28-model-cache" (OuterVolumeSpecName: "model-cache") pod "32403748-8961-4b62-81c8-81f9945d4f28" (UID: "32403748-8961-4b62-81c8-81f9945d4f28"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:19:15.387155 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.387132    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32403748-8961-4b62-81c8-81f9945d4f28-tmp-dir\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:19:15.387217 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.387153    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32403748-8961-4b62-81c8-81f9945d4f28-model-cache\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:19:15.387217 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.387166    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32403748-8961-4b62-81c8-81f9945d4f28-home\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:19:15.388996 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.388969    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32403748-8961-4b62-81c8-81f9945d4f28-dshm" (OuterVolumeSpecName: "dshm") pod "32403748-8961-4b62-81c8-81f9945d4f28" (UID: "32403748-8961-4b62-81c8-81f9945d4f28"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:19:15.389106 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.389054    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32403748-8961-4b62-81c8-81f9945d4f28-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32403748-8961-4b62-81c8-81f9945d4f28" (UID: "32403748-8961-4b62-81c8-81f9945d4f28"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:19:15.389106 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.389088    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32403748-8961-4b62-81c8-81f9945d4f28-kube-api-access-qsqfw" (OuterVolumeSpecName: "kube-api-access-qsqfw") pod "32403748-8961-4b62-81c8-81f9945d4f28" (UID: "32403748-8961-4b62-81c8-81f9945d4f28"). InnerVolumeSpecName "kube-api-access-qsqfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:19:15.487588 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.487555    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32403748-8961-4b62-81c8-81f9945d4f28-dshm\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:19:15.487588 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.487582    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32403748-8961-4b62-81c8-81f9945d4f28-tls-certs\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:19:15.487588 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:15.487592    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsqfw\" (UniqueName: \"kubernetes.io/projected/32403748-8961-4b62-81c8-81f9945d4f28-kube-api-access-qsqfw\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:19:16.123575 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:16.123539    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03f4c088-d3d6-48da-98af-deea6cba8145" path="/var/lib/kubelet/pods/03f4c088-d3d6-48da-98af-deea6cba8145/volumes"
Jul 27 19:19:16.165991 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:16.165952    2566 generic.go:356] "Generic (PLEG): container finished" podID="32403748-8961-4b62-81c8-81f9945d4f28" containerID="7c15b097604890dcc58324a287971bb3998f12babe805056daecb49d8cf05180" exitCode=0
Jul 27 19:19:16.166444 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:16.166012    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"
Jul 27 19:19:16.166444 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:16.166038    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf" event={"ID":"32403748-8961-4b62-81c8-81f9945d4f28","Type":"ContainerDied","Data":"7c15b097604890dcc58324a287971bb3998f12babe805056daecb49d8cf05180"}
Jul 27 19:19:16.166444 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:16.166083    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf" event={"ID":"32403748-8961-4b62-81c8-81f9945d4f28","Type":"ContainerDied","Data":"c2b80832865e5f98b41440dbb8df4ddba61d49da8b5343862d3718d1bc2635cb"}
Jul 27 19:19:16.166444 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:16.166112    2566 scope.go:117] "RemoveContainer" containerID="7c15b097604890dcc58324a287971bb3998f12babe805056daecb49d8cf05180"
Jul 27 19:19:16.175472 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:16.175453    2566 scope.go:117] "RemoveContainer" containerID="7c15b097604890dcc58324a287971bb3998f12babe805056daecb49d8cf05180"
Jul 27 19:19:16.175711 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:19:16.175691    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c15b097604890dcc58324a287971bb3998f12babe805056daecb49d8cf05180\": container with ID starting with 7c15b097604890dcc58324a287971bb3998f12babe805056daecb49d8cf05180 not found: ID does not exist" containerID="7c15b097604890dcc58324a287971bb3998f12babe805056daecb49d8cf05180"
Jul 27 19:19:16.175773 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:16.175722    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c15b097604890dcc58324a287971bb3998f12babe805056daecb49d8cf05180"} err="failed to get container status \"7c15b097604890dcc58324a287971bb3998f12babe805056daecb49d8cf05180\": rpc error: code = NotFound desc = could not find container \"7c15b097604890dcc58324a287971bb3998f12babe805056daecb49d8cf05180\": container with ID starting with 7c15b097604890dcc58324a287971bb3998f12babe805056daecb49d8cf05180 not found: ID does not exist"
Jul 27 19:19:16.188593 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:16.188567    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"]
Jul 27 19:19:16.192305 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:16.192283    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-6710a9/auth-test-model-6710a9-kserve-7bcf577699-4z7rf"]
Jul 27 19:19:18.124022 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:18.123993    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32403748-8961-4b62-81c8-81f9945d4f28" path="/var/lib/kubelet/pods/32403748-8961-4b62-81c8-81f9945d4f28/volumes"
Jul 27 19:19:50.124901 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.124826    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"]
Jul 27 19:19:50.125416 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.125235    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32403748-8961-4b62-81c8-81f9945d4f28" containerName="main"
Jul 27 19:19:50.125416 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.125249    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="32403748-8961-4b62-81c8-81f9945d4f28" containerName="main"
Jul 27 19:19:50.125416 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.125268    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03f4c088-d3d6-48da-98af-deea6cba8145" containerName="main"
Jul 27 19:19:50.125416 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.125273    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="03f4c088-d3d6-48da-98af-deea6cba8145" containerName="main"
Jul 27 19:19:50.125416 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.125330    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="03f4c088-d3d6-48da-98af-deea6cba8145" containerName="main"
Jul 27 19:19:50.125416 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.125341    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="32403748-8961-4b62-81c8-81f9945d4f28" containerName="main"
Jul 27 19:19:50.128180 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.128164    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"
Jul 27 19:19:50.133526 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.133505    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6710a9\"/\"kube-root-ca.crt\""
Jul 27 19:19:50.133658 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.133508    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6710a9\"/\"openshift-service-ca.crt\""
Jul 27 19:19:50.135054 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.135033    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6710a9\"/\"default-dockercfg-r8f6c\""
Jul 27 19:19:50.145031 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.145012    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6710a9\"/\"sub-test-model-6710a9-kserve-self-signed-certs\""
Jul 27 19:19:50.181559 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.181528    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"]
Jul 27 19:19:50.301760 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.301726    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-tmp-dir\") pod \"sub-test-model-6710a9-kserve-8685fd99b7-fhtpg\" (UID: \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\") " pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"
Jul 27 19:19:50.301943 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.301777    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x4hs\" (UniqueName: \"kubernetes.io/projected/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-kube-api-access-8x4hs\") pod \"sub-test-model-6710a9-kserve-8685fd99b7-fhtpg\" (UID: \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\") " pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"
Jul 27 19:19:50.301943 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.301823    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-model-cache\") pod \"sub-test-model-6710a9-kserve-8685fd99b7-fhtpg\" (UID: \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\") " pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"
Jul 27 19:19:50.301943 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.301874    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-tls-certs\") pod \"sub-test-model-6710a9-kserve-8685fd99b7-fhtpg\" (UID: \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\") " pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"
Jul 27 19:19:50.301943 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.301911    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-home\") pod \"sub-test-model-6710a9-kserve-8685fd99b7-fhtpg\" (UID: \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\") " pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"
Jul 27 19:19:50.302175 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.301949    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-dshm\") pod \"sub-test-model-6710a9-kserve-8685fd99b7-fhtpg\" (UID: \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\") " pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"
Jul 27 19:19:50.403593 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.403500    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-tls-certs\") pod \"sub-test-model-6710a9-kserve-8685fd99b7-fhtpg\" (UID: \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\") " pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"
Jul 27 19:19:50.403593 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.403557    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-home\") pod \"sub-test-model-6710a9-kserve-8685fd99b7-fhtpg\" (UID: \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\") " pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"
Jul 27 19:19:50.403832 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.403595    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-dshm\") pod \"sub-test-model-6710a9-kserve-8685fd99b7-fhtpg\" (UID: \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\") " pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"
Jul 27 19:19:50.403832 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.403639    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-tmp-dir\") pod \"sub-test-model-6710a9-kserve-8685fd99b7-fhtpg\" (UID: \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\") " pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"
Jul 27 19:19:50.403832 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.403674    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x4hs\" (UniqueName: \"kubernetes.io/projected/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-kube-api-access-8x4hs\") pod \"sub-test-model-6710a9-kserve-8685fd99b7-fhtpg\" (UID: \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\") " pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"
Jul 27 19:19:50.403832 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.403715    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-model-cache\") pod \"sub-test-model-6710a9-kserve-8685fd99b7-fhtpg\" (UID: \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\") " pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"
Jul 27 19:19:50.404058 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.404033    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-home\") pod \"sub-test-model-6710a9-kserve-8685fd99b7-fhtpg\" (UID: \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\") " pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"
Jul 27 19:19:50.404159 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.404036    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-tmp-dir\") pod \"sub-test-model-6710a9-kserve-8685fd99b7-fhtpg\" (UID: \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\") " pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"
Jul 27 19:19:50.404159 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.404101    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-model-cache\") pod \"sub-test-model-6710a9-kserve-8685fd99b7-fhtpg\" (UID: \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\") " pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"
Jul 27 19:19:50.405958 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.405940    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-dshm\") pod \"sub-test-model-6710a9-kserve-8685fd99b7-fhtpg\" (UID: \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\") " pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"
Jul 27 19:19:50.406257 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.406240    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-tls-certs\") pod \"sub-test-model-6710a9-kserve-8685fd99b7-fhtpg\" (UID: \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\") " pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"
Jul 27 19:19:50.428174 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.428139    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x4hs\" (UniqueName: \"kubernetes.io/projected/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-kube-api-access-8x4hs\") pod \"sub-test-model-6710a9-kserve-8685fd99b7-fhtpg\" (UID: \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\") " pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"
Jul 27 19:19:50.437141 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.437104    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"
Jul 27 19:19:50.585606 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:50.585581    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"]
Jul 27 19:19:50.586352 ip-10-0-128-196 kubenswrapper[2566]: W0727 19:19:50.586330    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb99e38ff_3ee6_4d2a_a5da_3c0b4054e82b.slice/crio-ee5323130525e80113613ffed4a2f4500caf3e19fb7c10c3c409eb76d0fd7b8a WatchSource:0}: Error finding container ee5323130525e80113613ffed4a2f4500caf3e19fb7c10c3c409eb76d0fd7b8a: Status 404 returned error can't find the container with id ee5323130525e80113613ffed4a2f4500caf3e19fb7c10c3c409eb76d0fd7b8a
Jul 27 19:19:51.359069 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:51.359031    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg" event={"ID":"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b","Type":"ContainerStarted","Data":"b651172ca4910777fb222252cf8919f81e2951eb06efd678cbff92325d9cd397"}
Jul 27 19:19:51.359069 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:51.359073    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg" event={"ID":"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b","Type":"ContainerStarted","Data":"ee5323130525e80113613ffed4a2f4500caf3e19fb7c10c3c409eb76d0fd7b8a"}
Jul 27 19:19:51.393913 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:19:51.393863    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg" podStartSLOduration=1.393848687 podStartE2EDuration="1.393848687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:19:51.389611444 +0000 UTC m=+3051.823869056" watchObservedRunningTime="2026-07-27 19:19:51.393848687 +0000 UTC m=+3051.828106297"
Jul 27 19:20:00.437824 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:20:00.437789    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"
Jul 27 19:20:00.438387 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:20:00.437987    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"
Jul 27 19:20:00.450449 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:20:00.450428    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"
Jul 27 19:20:01.429892 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:20:01.429862    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"
Jul 27 19:21:28.724563 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:28.724487    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"]
Jul 27 19:21:36.318719 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.318685    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"]
Jul 27 19:21:36.322379 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.322360    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"
Jul 27 19:21:36.325555 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.325533    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7f7473\"/\"default-dockercfg-5ncxf\""
Jul 27 19:21:36.325658 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.325573    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7f7473\"/\"kube-root-ca.crt\""
Jul 27 19:21:36.326905 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.326889    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7f7473\"/\"rate-test-model-7f7473-kserve-self-signed-certs\""
Jul 27 19:21:36.326969 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.326935    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7f7473\"/\"openshift-service-ca.crt\""
Jul 27 19:21:36.341803 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.341779    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"]
Jul 27 19:21:36.384813 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.384782    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxx8f\" (UniqueName: \"kubernetes.io/projected/f1d50505-5262-4b95-968b-15aacb712df0-kube-api-access-cxx8f\") pod \"rate-test-model-7f7473-kserve-f5f7bf5f-92t6s\" (UID: \"f1d50505-5262-4b95-968b-15aacb712df0\") " pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"
Jul 27 19:21:36.384963 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.384838    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1d50505-5262-4b95-968b-15aacb712df0-home\") pod \"rate-test-model-7f7473-kserve-f5f7bf5f-92t6s\" (UID: \"f1d50505-5262-4b95-968b-15aacb712df0\") " pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"
Jul 27 19:21:36.384963 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.384929    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1d50505-5262-4b95-968b-15aacb712df0-dshm\") pod \"rate-test-model-7f7473-kserve-f5f7bf5f-92t6s\" (UID: \"f1d50505-5262-4b95-968b-15aacb712df0\") " pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"
Jul 27 19:21:36.384963 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.384953    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1d50505-5262-4b95-968b-15aacb712df0-model-cache\") pod \"rate-test-model-7f7473-kserve-f5f7bf5f-92t6s\" (UID: \"f1d50505-5262-4b95-968b-15aacb712df0\") " pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"
Jul 27 19:21:36.385074 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.384972    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1d50505-5262-4b95-968b-15aacb712df0-tmp-dir\") pod \"rate-test-model-7f7473-kserve-f5f7bf5f-92t6s\" (UID: \"f1d50505-5262-4b95-968b-15aacb712df0\") " pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"
Jul 27 19:21:36.385074 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.384988    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1d50505-5262-4b95-968b-15aacb712df0-tls-certs\") pod \"rate-test-model-7f7473-kserve-f5f7bf5f-92t6s\" (UID: \"f1d50505-5262-4b95-968b-15aacb712df0\") " pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"
Jul 27 19:21:36.486041 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.486005    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxx8f\" (UniqueName: \"kubernetes.io/projected/f1d50505-5262-4b95-968b-15aacb712df0-kube-api-access-cxx8f\") pod \"rate-test-model-7f7473-kserve-f5f7bf5f-92t6s\" (UID: \"f1d50505-5262-4b95-968b-15aacb712df0\") " pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"
Jul 27 19:21:36.486296 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.486070    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1d50505-5262-4b95-968b-15aacb712df0-home\") pod \"rate-test-model-7f7473-kserve-f5f7bf5f-92t6s\" (UID: \"f1d50505-5262-4b95-968b-15aacb712df0\") " pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"
Jul 27 19:21:36.486296 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.486140    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1d50505-5262-4b95-968b-15aacb712df0-dshm\") pod \"rate-test-model-7f7473-kserve-f5f7bf5f-92t6s\" (UID: \"f1d50505-5262-4b95-968b-15aacb712df0\") " pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"
Jul 27 19:21:36.486296 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.486163    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1d50505-5262-4b95-968b-15aacb712df0-model-cache\") pod \"rate-test-model-7f7473-kserve-f5f7bf5f-92t6s\" (UID: \"f1d50505-5262-4b95-968b-15aacb712df0\") " pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"
Jul 27 19:21:36.486296 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.486198    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1d50505-5262-4b95-968b-15aacb712df0-tmp-dir\") pod \"rate-test-model-7f7473-kserve-f5f7bf5f-92t6s\" (UID: \"f1d50505-5262-4b95-968b-15aacb712df0\") " pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"
Jul 27 19:21:36.486296 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.486216    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1d50505-5262-4b95-968b-15aacb712df0-tls-certs\") pod \"rate-test-model-7f7473-kserve-f5f7bf5f-92t6s\" (UID: \"f1d50505-5262-4b95-968b-15aacb712df0\") " pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"
Jul 27 19:21:36.486603 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.486580    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1d50505-5262-4b95-968b-15aacb712df0-home\") pod \"rate-test-model-7f7473-kserve-f5f7bf5f-92t6s\" (UID: \"f1d50505-5262-4b95-968b-15aacb712df0\") " pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"
Jul 27 19:21:36.486663 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.486585    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1d50505-5262-4b95-968b-15aacb712df0-model-cache\") pod \"rate-test-model-7f7473-kserve-f5f7bf5f-92t6s\" (UID: \"f1d50505-5262-4b95-968b-15aacb712df0\") " pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"
Jul 27 19:21:36.486663 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.486651    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1d50505-5262-4b95-968b-15aacb712df0-tmp-dir\") pod \"rate-test-model-7f7473-kserve-f5f7bf5f-92t6s\" (UID: \"f1d50505-5262-4b95-968b-15aacb712df0\") " pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"
Jul 27 19:21:36.488644 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.488625    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1d50505-5262-4b95-968b-15aacb712df0-dshm\") pod \"rate-test-model-7f7473-kserve-f5f7bf5f-92t6s\" (UID: \"f1d50505-5262-4b95-968b-15aacb712df0\") " pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"
Jul 27 19:21:36.488821 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.488806    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1d50505-5262-4b95-968b-15aacb712df0-tls-certs\") pod \"rate-test-model-7f7473-kserve-f5f7bf5f-92t6s\" (UID: \"f1d50505-5262-4b95-968b-15aacb712df0\") " pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"
Jul 27 19:21:36.495287 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.495268    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxx8f\" (UniqueName: \"kubernetes.io/projected/f1d50505-5262-4b95-968b-15aacb712df0-kube-api-access-cxx8f\") pod \"rate-test-model-7f7473-kserve-f5f7bf5f-92t6s\" (UID: \"f1d50505-5262-4b95-968b-15aacb712df0\") " pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"
Jul 27 19:21:36.631359 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.631264    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"
Jul 27 19:21:36.787893 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.787865    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"]
Jul 27 19:21:36.789380 ip-10-0-128-196 kubenswrapper[2566]: W0727 19:21:36.789339    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1d50505_5262_4b95_968b_15aacb712df0.slice/crio-bd77e3d7ca0fb8067d98d794064096c448467e1035b9a8123d3239349492d35a WatchSource:0}: Error finding container bd77e3d7ca0fb8067d98d794064096c448467e1035b9a8123d3239349492d35a: Status 404 returned error can't find the container with id bd77e3d7ca0fb8067d98d794064096c448467e1035b9a8123d3239349492d35a
Jul 27 19:21:36.948222 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.948136    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s" event={"ID":"f1d50505-5262-4b95-968b-15aacb712df0","Type":"ContainerStarted","Data":"f96cf412976bf6df8b66d24e60a11a2fdc9d1169f83d28679d2abbcf841cf61c"}
Jul 27 19:21:36.948222 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.948186    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s" event={"ID":"f1d50505-5262-4b95-968b-15aacb712df0","Type":"ContainerStarted","Data":"bd77e3d7ca0fb8067d98d794064096c448467e1035b9a8123d3239349492d35a"}
Jul 27 19:21:36.972290 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:36.972232    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s" podStartSLOduration=0.972213219 podStartE2EDuration="972.213219ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:21:36.971338556 +0000 UTC m=+3157.405596167" watchObservedRunningTime="2026-07-27 19:21:36.972213219 +0000 UTC m=+3157.406470829"
Jul 27 19:21:43.744107 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:43.744048    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg" podUID="b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b" containerName="main" containerID="cri-o://b651172ca4910777fb222252cf8919f81e2951eb06efd678cbff92325d9cd397" gracePeriod=45
Jul 27 19:21:43.988514 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:43.988485    2566 generic.go:356] "Generic (PLEG): container finished" podID="b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b" containerID="b651172ca4910777fb222252cf8919f81e2951eb06efd678cbff92325d9cd397" exitCode=0
Jul 27 19:21:43.988667 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:43.988568    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg" event={"ID":"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b","Type":"ContainerDied","Data":"b651172ca4910777fb222252cf8919f81e2951eb06efd678cbff92325d9cd397"}
Jul 27 19:21:43.988667 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:43.988632    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg" event={"ID":"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b","Type":"ContainerDied","Data":"ee5323130525e80113613ffed4a2f4500caf3e19fb7c10c3c409eb76d0fd7b8a"}
Jul 27 19:21:43.988667 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:43.988651    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee5323130525e80113613ffed4a2f4500caf3e19fb7c10c3c409eb76d0fd7b8a"
Jul 27 19:21:44.003674 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:44.003621    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"
Jul 27 19:21:44.060284 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:44.060254    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-home\") pod \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\" (UID: \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\") "
Jul 27 19:21:44.060455 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:44.060332    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-tmp-dir\") pod \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\" (UID: \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\") "
Jul 27 19:21:44.060455 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:44.060356    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8x4hs\" (UniqueName: \"kubernetes.io/projected/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-kube-api-access-8x4hs\") pod \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\" (UID: \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\") "
Jul 27 19:21:44.060455 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:44.060378    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-model-cache\") pod \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\" (UID: \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\") "
Jul 27 19:21:44.060455 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:44.060410    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-dshm\") pod \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\" (UID: \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\") "
Jul 27 19:21:44.060673 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:44.060555    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-home" (OuterVolumeSpecName: "home") pod "b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b" (UID: "b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:21:44.060673 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:44.060564    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-tls-certs\") pod \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\" (UID: \"b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b\") "
Jul 27 19:21:44.060673 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:44.060664    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-model-cache" (OuterVolumeSpecName: "model-cache") pod "b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b" (UID: "b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:21:44.060811 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:44.060679    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b" (UID: "b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:21:44.060916 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:44.060899    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-home\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:21:44.060954 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:44.060924    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-tmp-dir\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:21:44.060954 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:44.060939    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-model-cache\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:21:44.062674 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:44.062654    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-dshm" (OuterVolumeSpecName: "dshm") pod "b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b" (UID: "b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:21:44.062774 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:44.062711    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b" (UID: "b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:21:44.062774 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:44.062740    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-kube-api-access-8x4hs" (OuterVolumeSpecName: "kube-api-access-8x4hs") pod "b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b" (UID: "b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b"). InnerVolumeSpecName "kube-api-access-8x4hs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:21:44.162248 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:44.162218    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x4hs\" (UniqueName: \"kubernetes.io/projected/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-kube-api-access-8x4hs\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:21:44.162248 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:44.162243    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-dshm\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:21:44.162248 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:44.162252    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b-tls-certs\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:21:44.993936 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:44.993909    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"
Jul 27 19:21:45.015198 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:45.015160    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"]
Jul 27 19:21:45.020940 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:45.020907    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-6710a9/sub-test-model-6710a9-kserve-8685fd99b7-fhtpg"]
Jul 27 19:21:46.123827 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:46.123795    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b" path="/var/lib/kubelet/pods/b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b/volumes"
Jul 27 19:21:46.632080 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:46.632044    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"
Jul 27 19:21:46.632333 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:46.632150    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"
Jul 27 19:21:46.645191 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:46.645167    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"
Jul 27 19:21:47.017259 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:21:47.017179    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"
Jul 27 19:23:15.158001 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:15.157967    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"]
Jul 27 19:23:17.161851 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:17.161813    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7701-140320380181040"]
Jul 27 19:23:17.162288 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:17.162274    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b" containerName="main"
Jul 27 19:23:17.162337 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:17.162290    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b" containerName="main"
Jul 27 19:23:17.162389 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:17.162379    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b99e38ff-3ee6-4d2a-a5da-3c0b4054e82b" containerName="main"
Jul 27 19:23:17.165507 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:17.165488    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-140320380181040"
Jul 27 19:23:17.169598 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:17.169571    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5jdw9\""
Jul 27 19:23:17.182583 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:17.182553    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7701-140320380181040"]
Jul 27 19:23:17.218971 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:17.218936    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8zxp\" (UniqueName: \"kubernetes.io/projected/cc0e8221-76af-43ff-90cc-b83ec2a7479a-kube-api-access-l8zxp\") pod \"test-curl-7701-140320380181040\" (UID: \"cc0e8221-76af-43ff-90cc-b83ec2a7479a\") " pod="opendatahub/test-curl-7701-140320380181040"
Jul 27 19:23:17.320375 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:17.320337    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8zxp\" (UniqueName: \"kubernetes.io/projected/cc0e8221-76af-43ff-90cc-b83ec2a7479a-kube-api-access-l8zxp\") pod \"test-curl-7701-140320380181040\" (UID: \"cc0e8221-76af-43ff-90cc-b83ec2a7479a\") " pod="opendatahub/test-curl-7701-140320380181040"
Jul 27 19:23:17.342052 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:17.342029    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8zxp\" (UniqueName: \"kubernetes.io/projected/cc0e8221-76af-43ff-90cc-b83ec2a7479a-kube-api-access-l8zxp\") pod \"test-curl-7701-140320380181040\" (UID: \"cc0e8221-76af-43ff-90cc-b83ec2a7479a\") " pod="opendatahub/test-curl-7701-140320380181040"
Jul 27 19:23:17.475070 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:17.474980    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-140320380181040"
Jul 27 19:23:17.636585 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:17.634777    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7701-140320380181040"]
Jul 27 19:23:17.638067 ip-10-0-128-196 kubenswrapper[2566]: W0727 19:23:17.638037    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc0e8221_76af_43ff_90cc_b83ec2a7479a.slice/crio-ad5d04a15a5b7eb89f17e6b07e8e9421506bda93479231d193b386d7216809b4 WatchSource:0}: Error finding container ad5d04a15a5b7eb89f17e6b07e8e9421506bda93479231d193b386d7216809b4: Status 404 returned error can't find the container with id ad5d04a15a5b7eb89f17e6b07e8e9421506bda93479231d193b386d7216809b4
Jul 27 19:23:17.639306 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:17.639289    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 19:23:18.514667 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:18.514630    2566 generic.go:356] "Generic (PLEG): container finished" podID="cc0e8221-76af-43ff-90cc-b83ec2a7479a" containerID="172e6b9e3e9d745325a714ba04f2cee8bbfe26831f88363a6360f3127728c238" exitCode=0
Jul 27 19:23:18.515016 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:18.514701    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7701-140320380181040" event={"ID":"cc0e8221-76af-43ff-90cc-b83ec2a7479a","Type":"ContainerDied","Data":"172e6b9e3e9d745325a714ba04f2cee8bbfe26831f88363a6360f3127728c238"}
Jul 27 19:23:18.515016 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:18.514728    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7701-140320380181040" event={"ID":"cc0e8221-76af-43ff-90cc-b83ec2a7479a","Type":"ContainerStarted","Data":"ad5d04a15a5b7eb89f17e6b07e8e9421506bda93479231d193b386d7216809b4"}
Jul 27 19:23:19.655361 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:19.655333    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-140320380181040"
Jul 27 19:23:19.744096 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:19.744067    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8zxp\" (UniqueName: \"kubernetes.io/projected/cc0e8221-76af-43ff-90cc-b83ec2a7479a-kube-api-access-l8zxp\") pod \"cc0e8221-76af-43ff-90cc-b83ec2a7479a\" (UID: \"cc0e8221-76af-43ff-90cc-b83ec2a7479a\") "
Jul 27 19:23:19.746317 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:19.746294    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0e8221-76af-43ff-90cc-b83ec2a7479a-kube-api-access-l8zxp" (OuterVolumeSpecName: "kube-api-access-l8zxp") pod "cc0e8221-76af-43ff-90cc-b83ec2a7479a" (UID: "cc0e8221-76af-43ff-90cc-b83ec2a7479a"). InnerVolumeSpecName "kube-api-access-l8zxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:23:19.754754 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:19.754731    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7701-140320380181040_cc0e8221-76af-43ff-90cc-b83ec2a7479a/test-curl-7701-140320380181040/0.log"
Jul 27 19:23:19.803730 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:19.803697    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7701-140320380181040"]
Jul 27 19:23:19.805961 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:19.805936    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7701-140320380181040"]
Jul 27 19:23:19.845544 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:19.845517    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8zxp\" (UniqueName: \"kubernetes.io/projected/cc0e8221-76af-43ff-90cc-b83ec2a7479a-kube-api-access-l8zxp\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:23:19.972338 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:19.972300    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7701-140320378406704"]
Jul 27 19:23:19.972744 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:19.972730    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc0e8221-76af-43ff-90cc-b83ec2a7479a" containerName="test-curl-7701-140320380181040"
Jul 27 19:23:19.972794 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:19.972747    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc0e8221-76af-43ff-90cc-b83ec2a7479a" containerName="test-curl-7701-140320380181040"
Jul 27 19:23:19.972837 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:19.972827    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc0e8221-76af-43ff-90cc-b83ec2a7479a" containerName="test-curl-7701-140320380181040"
Jul 27 19:23:19.975923 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:19.975905    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-140320378406704"
Jul 27 19:23:19.990733 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:19.990704    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7701-140320378406704"]
Jul 27 19:23:20.047255 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:20.047220    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxv7j\" (UniqueName: \"kubernetes.io/projected/741f68df-0be2-4b06-9dbe-f0442c039878-kube-api-access-qxv7j\") pod \"test-curl-7701-140320378406704\" (UID: \"741f68df-0be2-4b06-9dbe-f0442c039878\") " pod="opendatahub/test-curl-7701-140320378406704"
Jul 27 19:23:20.123861 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:20.123785    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc0e8221-76af-43ff-90cc-b83ec2a7479a" path="/var/lib/kubelet/pods/cc0e8221-76af-43ff-90cc-b83ec2a7479a/volumes"
Jul 27 19:23:20.148094 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:20.148053    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxv7j\" (UniqueName: \"kubernetes.io/projected/741f68df-0be2-4b06-9dbe-f0442c039878-kube-api-access-qxv7j\") pod \"test-curl-7701-140320378406704\" (UID: \"741f68df-0be2-4b06-9dbe-f0442c039878\") " pod="opendatahub/test-curl-7701-140320378406704"
Jul 27 19:23:20.165374 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:20.165348    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxv7j\" (UniqueName: \"kubernetes.io/projected/741f68df-0be2-4b06-9dbe-f0442c039878-kube-api-access-qxv7j\") pod \"test-curl-7701-140320378406704\" (UID: \"741f68df-0be2-4b06-9dbe-f0442c039878\") " pod="opendatahub/test-curl-7701-140320378406704"
Jul 27 19:23:20.285009 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:20.284978    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-140320378406704"
Jul 27 19:23:20.426866 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:20.426835    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7701-140320378406704"]
Jul 27 19:23:20.428903 ip-10-0-128-196 kubenswrapper[2566]: W0727 19:23:20.428874    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod741f68df_0be2_4b06_9dbe_f0442c039878.slice/crio-1452529e0abfcfe46c050e4f3cefacdc5331d71451694038539d03ca4cc5bec0 WatchSource:0}: Error finding container 1452529e0abfcfe46c050e4f3cefacdc5331d71451694038539d03ca4cc5bec0: Status 404 returned error can't find the container with id 1452529e0abfcfe46c050e4f3cefacdc5331d71451694038539d03ca4cc5bec0
Jul 27 19:23:20.526611 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:20.526580    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-140320380181040"
Jul 27 19:23:20.526805 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:20.526637    2566 scope.go:117] "RemoveContainer" containerID="172e6b9e3e9d745325a714ba04f2cee8bbfe26831f88363a6360f3127728c238"
Jul 27 19:23:20.527972 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:20.527850    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7701-140320378406704" event={"ID":"741f68df-0be2-4b06-9dbe-f0442c039878","Type":"ContainerStarted","Data":"1452529e0abfcfe46c050e4f3cefacdc5331d71451694038539d03ca4cc5bec0"}
Jul 27 19:23:21.534691 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:21.534655    2566 generic.go:356] "Generic (PLEG): container finished" podID="741f68df-0be2-4b06-9dbe-f0442c039878" containerID="794ac7960f47afd6d2028df87e32ea87cd32c4d55c26849ebf04658db5b282db" exitCode=0
Jul 27 19:23:21.535147 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:21.534738    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7701-140320378406704" event={"ID":"741f68df-0be2-4b06-9dbe-f0442c039878","Type":"ContainerDied","Data":"794ac7960f47afd6d2028df87e32ea87cd32c4d55c26849ebf04658db5b282db"}
Jul 27 19:23:22.682998 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:22.682970    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-140320378406704"
Jul 27 19:23:22.720406 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:22.720304    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7701-140320378406704_741f68df-0be2-4b06-9dbe-f0442c039878/test-curl-7701-140320378406704/0.log"
Jul 27 19:23:22.753924 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:22.753889    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7701-140320378406704"]
Jul 27 19:23:22.758781 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:22.758748    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7701-140320378406704"]
Jul 27 19:23:22.774312 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:22.774278    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxv7j\" (UniqueName: \"kubernetes.io/projected/741f68df-0be2-4b06-9dbe-f0442c039878-kube-api-access-qxv7j\") pod \"741f68df-0be2-4b06-9dbe-f0442c039878\" (UID: \"741f68df-0be2-4b06-9dbe-f0442c039878\") "
Jul 27 19:23:22.776585 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:22.776549    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741f68df-0be2-4b06-9dbe-f0442c039878-kube-api-access-qxv7j" (OuterVolumeSpecName: "kube-api-access-qxv7j") pod "741f68df-0be2-4b06-9dbe-f0442c039878" (UID: "741f68df-0be2-4b06-9dbe-f0442c039878"). InnerVolumeSpecName "kube-api-access-qxv7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:23:22.876062 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:22.875981    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxv7j\" (UniqueName: \"kubernetes.io/projected/741f68df-0be2-4b06-9dbe-f0442c039878-kube-api-access-qxv7j\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:23:22.946529 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:22.946493    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7701-140320378423312"]
Jul 27 19:23:22.946911 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:22.946898    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="741f68df-0be2-4b06-9dbe-f0442c039878" containerName="test-curl-7701-140320378406704"
Jul 27 19:23:22.946957 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:22.946913    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="741f68df-0be2-4b06-9dbe-f0442c039878" containerName="test-curl-7701-140320378406704"
Jul 27 19:23:22.946993 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:22.946971    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="741f68df-0be2-4b06-9dbe-f0442c039878" containerName="test-curl-7701-140320378406704"
Jul 27 19:23:22.950113 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:22.950083    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-140320378423312"
Jul 27 19:23:22.961232 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:22.961206    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7701-140320378423312"]
Jul 27 19:23:23.077767 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:23.077728    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mch6j\" (UniqueName: \"kubernetes.io/projected/6a7dddd8-e0f4-4e39-af92-bc69ee7f7482-kube-api-access-mch6j\") pod \"test-curl-7701-140320378423312\" (UID: \"6a7dddd8-e0f4-4e39-af92-bc69ee7f7482\") " pod="opendatahub/test-curl-7701-140320378423312"
Jul 27 19:23:23.179273 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:23.178827    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mch6j\" (UniqueName: \"kubernetes.io/projected/6a7dddd8-e0f4-4e39-af92-bc69ee7f7482-kube-api-access-mch6j\") pod \"test-curl-7701-140320378423312\" (UID: \"6a7dddd8-e0f4-4e39-af92-bc69ee7f7482\") " pod="opendatahub/test-curl-7701-140320378423312"
Jul 27 19:23:23.189376 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:23.189355    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mch6j\" (UniqueName: \"kubernetes.io/projected/6a7dddd8-e0f4-4e39-af92-bc69ee7f7482-kube-api-access-mch6j\") pod \"test-curl-7701-140320378423312\" (UID: \"6a7dddd8-e0f4-4e39-af92-bc69ee7f7482\") " pod="opendatahub/test-curl-7701-140320378423312"
Jul 27 19:23:23.259359 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:23.259318    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-140320378423312"
Jul 27 19:23:23.403915 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:23.403890    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7701-140320378423312"]
Jul 27 19:23:23.405933 ip-10-0-128-196 kubenswrapper[2566]: W0727 19:23:23.405908    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a7dddd8_e0f4_4e39_af92_bc69ee7f7482.slice/crio-284602a5a427feecf3c8f345168a2ac8056b023ca202097eace1937e71cb2aae WatchSource:0}: Error finding container 284602a5a427feecf3c8f345168a2ac8056b023ca202097eace1937e71cb2aae: Status 404 returned error can't find the container with id 284602a5a427feecf3c8f345168a2ac8056b023ca202097eace1937e71cb2aae
Jul 27 19:23:23.548418 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:23.548383    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7701-140320378423312" event={"ID":"6a7dddd8-e0f4-4e39-af92-bc69ee7f7482","Type":"ContainerStarted","Data":"284602a5a427feecf3c8f345168a2ac8056b023ca202097eace1937e71cb2aae"}
Jul 27 19:23:23.549877 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:23.549853    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1452529e0abfcfe46c050e4f3cefacdc5331d71451694038539d03ca4cc5bec0"
Jul 27 19:23:23.550002 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:23.549909    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-140320378406704"
Jul 27 19:23:23.560727 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:23:23.560699    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7701-140320378406704\" is forbidden: User \"system:node:ip-10-0-128-196.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-196.ec2.internal' and this object" podUID="741f68df-0be2-4b06-9dbe-f0442c039878" pod="opendatahub/test-curl-7701-140320378406704"
Jul 27 19:23:24.124078 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:24.124043    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="741f68df-0be2-4b06-9dbe-f0442c039878" path="/var/lib/kubelet/pods/741f68df-0be2-4b06-9dbe-f0442c039878/volumes"
Jul 27 19:23:24.557407 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:24.557375    2566 generic.go:356] "Generic (PLEG): container finished" podID="6a7dddd8-e0f4-4e39-af92-bc69ee7f7482" containerID="001067ca3993f132acc5ce02d3abd858ab19ab534a8866da5332b0c31453768c" exitCode=0
Jul 27 19:23:24.557562 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:24.557415    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7701-140320378423312" event={"ID":"6a7dddd8-e0f4-4e39-af92-bc69ee7f7482","Type":"ContainerDied","Data":"001067ca3993f132acc5ce02d3abd858ab19ab534a8866da5332b0c31453768c"}
Jul 27 19:23:25.703437 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:25.703411    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-140320378423312"
Jul 27 19:23:25.728784 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:25.728760    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7701-140320378423312_6a7dddd8-e0f4-4e39-af92-bc69ee7f7482/test-curl-7701-140320378423312/0.log"
Jul 27 19:23:25.761902 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:25.761876    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7701-140320378423312"]
Jul 27 19:23:25.764153 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:25.764113    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7701-140320378423312"]
Jul 27 19:23:25.804693 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:25.804659    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mch6j\" (UniqueName: \"kubernetes.io/projected/6a7dddd8-e0f4-4e39-af92-bc69ee7f7482-kube-api-access-mch6j\") pod \"6a7dddd8-e0f4-4e39-af92-bc69ee7f7482\" (UID: \"6a7dddd8-e0f4-4e39-af92-bc69ee7f7482\") "
Jul 27 19:23:25.806882 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:25.806854    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7dddd8-e0f4-4e39-af92-bc69ee7f7482-kube-api-access-mch6j" (OuterVolumeSpecName: "kube-api-access-mch6j") pod "6a7dddd8-e0f4-4e39-af92-bc69ee7f7482" (UID: "6a7dddd8-e0f4-4e39-af92-bc69ee7f7482"). InnerVolumeSpecName "kube-api-access-mch6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:23:25.893520 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:25.893436    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7701-140320378423312"]
Jul 27 19:23:25.893858 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:25.893844    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a7dddd8-e0f4-4e39-af92-bc69ee7f7482" containerName="test-curl-7701-140320378423312"
Jul 27 19:23:25.893897 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:25.893860    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a7dddd8-e0f4-4e39-af92-bc69ee7f7482" containerName="test-curl-7701-140320378423312"
Jul 27 19:23:25.893932 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:25.893917    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a7dddd8-e0f4-4e39-af92-bc69ee7f7482" containerName="test-curl-7701-140320378423312"
Jul 27 19:23:25.896957 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:25.896941    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-140320378423312"
Jul 27 19:23:25.902355 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:25.902333    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7701-140320378423312"]
Jul 27 19:23:25.905490 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:25.905462    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mch6j\" (UniqueName: \"kubernetes.io/projected/6a7dddd8-e0f4-4e39-af92-bc69ee7f7482-kube-api-access-mch6j\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:23:26.006901 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:26.006861    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4nts\" (UniqueName: \"kubernetes.io/projected/1e39037f-2959-44a4-b504-1fcaccf3e01b-kube-api-access-h4nts\") pod \"test-curl-7701-140320378423312\" (UID: \"1e39037f-2959-44a4-b504-1fcaccf3e01b\") " pod="opendatahub/test-curl-7701-140320378423312"
Jul 27 19:23:26.107721 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:26.107686    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4nts\" (UniqueName: \"kubernetes.io/projected/1e39037f-2959-44a4-b504-1fcaccf3e01b-kube-api-access-h4nts\") pod \"test-curl-7701-140320378423312\" (UID: \"1e39037f-2959-44a4-b504-1fcaccf3e01b\") " pod="opendatahub/test-curl-7701-140320378423312"
Jul 27 19:23:26.119679 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:26.119641    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4nts\" (UniqueName: \"kubernetes.io/projected/1e39037f-2959-44a4-b504-1fcaccf3e01b-kube-api-access-h4nts\") pod \"test-curl-7701-140320378423312\" (UID: \"1e39037f-2959-44a4-b504-1fcaccf3e01b\") " pod="opendatahub/test-curl-7701-140320378423312"
Jul 27 19:23:26.124219 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:26.124191    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a7dddd8-e0f4-4e39-af92-bc69ee7f7482" path="/var/lib/kubelet/pods/6a7dddd8-e0f4-4e39-af92-bc69ee7f7482/volumes"
Jul 27 19:23:26.207651 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:26.207559    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-140320378423312"
Jul 27 19:23:26.560739 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:26.560714    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7701-140320378423312"]
Jul 27 19:23:26.562620 ip-10-0-128-196 kubenswrapper[2566]: W0727 19:23:26.562593    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e39037f_2959_44a4_b504_1fcaccf3e01b.slice/crio-5fb726eeb53acee0ee863f39db7f4a8dbae01a84f8142056f80fa71230a26d7a WatchSource:0}: Error finding container 5fb726eeb53acee0ee863f39db7f4a8dbae01a84f8142056f80fa71230a26d7a: Status 404 returned error can't find the container with id 5fb726eeb53acee0ee863f39db7f4a8dbae01a84f8142056f80fa71230a26d7a
Jul 27 19:23:26.568945 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:26.568927    2566 scope.go:117] "RemoveContainer" containerID="001067ca3993f132acc5ce02d3abd858ab19ab534a8866da5332b0c31453768c"
Jul 27 19:23:26.569031 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:26.568935    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-140320378423312"
Jul 27 19:23:27.578000 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:27.577963    2566 generic.go:356] "Generic (PLEG): container finished" podID="1e39037f-2959-44a4-b504-1fcaccf3e01b" containerID="97d3c7a41dc39a9518cb2354b0449bbc1f23f5285294b063f952ed3186b9cc70" exitCode=0
Jul 27 19:23:27.578421 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:27.578024    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7701-140320378423312" event={"ID":"1e39037f-2959-44a4-b504-1fcaccf3e01b","Type":"ContainerDied","Data":"97d3c7a41dc39a9518cb2354b0449bbc1f23f5285294b063f952ed3186b9cc70"}
Jul 27 19:23:27.578421 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:27.578058    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7701-140320378423312" event={"ID":"1e39037f-2959-44a4-b504-1fcaccf3e01b","Type":"ContainerStarted","Data":"5fb726eeb53acee0ee863f39db7f4a8dbae01a84f8142056f80fa71230a26d7a"}
Jul 27 19:23:28.721726 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:28.721698    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-140320378423312"
Jul 27 19:23:28.763501 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:28.763472    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7701-140320378423312_1e39037f-2959-44a4-b504-1fcaccf3e01b/test-curl-7701-140320378423312/0.log"
Jul 27 19:23:28.817226 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:28.817181    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7701-140320378423312"]
Jul 27 19:23:28.820797 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:28.820773    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7701-140320378423312"]
Jul 27 19:23:28.834622 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:28.834559    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4nts\" (UniqueName: \"kubernetes.io/projected/1e39037f-2959-44a4-b504-1fcaccf3e01b-kube-api-access-h4nts\") pod \"1e39037f-2959-44a4-b504-1fcaccf3e01b\" (UID: \"1e39037f-2959-44a4-b504-1fcaccf3e01b\") "
Jul 27 19:23:28.836754 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:28.836732    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e39037f-2959-44a4-b504-1fcaccf3e01b-kube-api-access-h4nts" (OuterVolumeSpecName: "kube-api-access-h4nts") pod "1e39037f-2959-44a4-b504-1fcaccf3e01b" (UID: "1e39037f-2959-44a4-b504-1fcaccf3e01b"). InnerVolumeSpecName "kube-api-access-h4nts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:23:28.935816 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:28.935787    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4nts\" (UniqueName: \"kubernetes.io/projected/1e39037f-2959-44a4-b504-1fcaccf3e01b-kube-api-access-h4nts\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:23:28.952653 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:28.952618    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7701-140320378424320"]
Jul 27 19:23:28.953027 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:28.953014    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e39037f-2959-44a4-b504-1fcaccf3e01b" containerName="test-curl-7701-140320378423312"
Jul 27 19:23:28.953070 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:28.953030    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e39037f-2959-44a4-b504-1fcaccf3e01b" containerName="test-curl-7701-140320378423312"
Jul 27 19:23:28.953136 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:28.953127    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e39037f-2959-44a4-b504-1fcaccf3e01b" containerName="test-curl-7701-140320378423312"
Jul 27 19:23:28.956269 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:28.956254    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-140320378424320"
Jul 27 19:23:28.964769 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:28.964748    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7701-140320378424320"]
Jul 27 19:23:29.042152 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:29.037325    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qqpm\" (UniqueName: \"kubernetes.io/projected/627605e0-2df0-40ee-a816-62ab2219ff69-kube-api-access-5qqpm\") pod \"test-curl-7701-140320378424320\" (UID: \"627605e0-2df0-40ee-a816-62ab2219ff69\") " pod="opendatahub/test-curl-7701-140320378424320"
Jul 27 19:23:29.138598 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:29.138495    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qqpm\" (UniqueName: \"kubernetes.io/projected/627605e0-2df0-40ee-a816-62ab2219ff69-kube-api-access-5qqpm\") pod \"test-curl-7701-140320378424320\" (UID: \"627605e0-2df0-40ee-a816-62ab2219ff69\") " pod="opendatahub/test-curl-7701-140320378424320"
Jul 27 19:23:29.153878 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:29.153846    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qqpm\" (UniqueName: \"kubernetes.io/projected/627605e0-2df0-40ee-a816-62ab2219ff69-kube-api-access-5qqpm\") pod \"test-curl-7701-140320378424320\" (UID: \"627605e0-2df0-40ee-a816-62ab2219ff69\") " pod="opendatahub/test-curl-7701-140320378424320"
Jul 27 19:23:29.266412 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:29.266376    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-140320378424320"
Jul 27 19:23:29.419016 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:29.418993    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7701-140320378424320"]
Jul 27 19:23:29.420685 ip-10-0-128-196 kubenswrapper[2566]: W0727 19:23:29.420652    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod627605e0_2df0_40ee_a816_62ab2219ff69.slice/crio-764237edd1c1e640effd45b37b5444424b89b8af838daac87683ce06393a10a5 WatchSource:0}: Error finding container 764237edd1c1e640effd45b37b5444424b89b8af838daac87683ce06393a10a5: Status 404 returned error can't find the container with id 764237edd1c1e640effd45b37b5444424b89b8af838daac87683ce06393a10a5
Jul 27 19:23:29.591739 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:29.591700    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7701-140320378424320" event={"ID":"627605e0-2df0-40ee-a816-62ab2219ff69","Type":"ContainerStarted","Data":"764237edd1c1e640effd45b37b5444424b89b8af838daac87683ce06393a10a5"}
Jul 27 19:23:29.592980 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:29.592960    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fb726eeb53acee0ee863f39db7f4a8dbae01a84f8142056f80fa71230a26d7a"
Jul 27 19:23:29.593151 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:29.592997    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-140320378423312"
Jul 27 19:23:29.605532 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:23:29.605506    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7701-140320378423312\" is forbidden: User \"system:node:ip-10-0-128-196.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-196.ec2.internal' and this object" podUID="1e39037f-2959-44a4-b504-1fcaccf3e01b" pod="opendatahub/test-curl-7701-140320378423312"
Jul 27 19:23:30.123589 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:23:30.123555    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7701-140320378423312\" is forbidden: User \"system:node:ip-10-0-128-196.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-196.ec2.internal' and this object" podUID="1e39037f-2959-44a4-b504-1fcaccf3e01b" pod="opendatahub/test-curl-7701-140320378423312"
Jul 27 19:23:30.125600 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.125574    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e39037f-2959-44a4-b504-1fcaccf3e01b" path="/var/lib/kubelet/pods/1e39037f-2959-44a4-b504-1fcaccf3e01b/volumes"
Jul 27 19:23:30.177535 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.177500    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s" podUID="f1d50505-5262-4b95-968b-15aacb712df0" containerName="main" containerID="cri-o://f96cf412976bf6df8b66d24e60a11a2fdc9d1169f83d28679d2abbcf841cf61c" gracePeriod=45
Jul 27 19:23:30.429722 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.429694    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"
Jul 27 19:23:30.552838 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.552803    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1d50505-5262-4b95-968b-15aacb712df0-home\") pod \"f1d50505-5262-4b95-968b-15aacb712df0\" (UID: \"f1d50505-5262-4b95-968b-15aacb712df0\") "
Jul 27 19:23:30.553040 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.552913    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxx8f\" (UniqueName: \"kubernetes.io/projected/f1d50505-5262-4b95-968b-15aacb712df0-kube-api-access-cxx8f\") pod \"f1d50505-5262-4b95-968b-15aacb712df0\" (UID: \"f1d50505-5262-4b95-968b-15aacb712df0\") "
Jul 27 19:23:30.553040 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.552932    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1d50505-5262-4b95-968b-15aacb712df0-dshm\") pod \"f1d50505-5262-4b95-968b-15aacb712df0\" (UID: \"f1d50505-5262-4b95-968b-15aacb712df0\") "
Jul 27 19:23:30.553040 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.552949    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1d50505-5262-4b95-968b-15aacb712df0-model-cache\") pod \"f1d50505-5262-4b95-968b-15aacb712df0\" (UID: \"f1d50505-5262-4b95-968b-15aacb712df0\") "
Jul 27 19:23:30.553040 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.552972    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1d50505-5262-4b95-968b-15aacb712df0-tls-certs\") pod \"f1d50505-5262-4b95-968b-15aacb712df0\" (UID: \"f1d50505-5262-4b95-968b-15aacb712df0\") "
Jul 27 19:23:30.553040 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.552996    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1d50505-5262-4b95-968b-15aacb712df0-tmp-dir\") pod \"f1d50505-5262-4b95-968b-15aacb712df0\" (UID: \"f1d50505-5262-4b95-968b-15aacb712df0\") "
Jul 27 19:23:30.553340 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.553167    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d50505-5262-4b95-968b-15aacb712df0-home" (OuterVolumeSpecName: "home") pod "f1d50505-5262-4b95-968b-15aacb712df0" (UID: "f1d50505-5262-4b95-968b-15aacb712df0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:23:30.553340 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.553288    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d50505-5262-4b95-968b-15aacb712df0-model-cache" (OuterVolumeSpecName: "model-cache") pod "f1d50505-5262-4b95-968b-15aacb712df0" (UID: "f1d50505-5262-4b95-968b-15aacb712df0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:23:30.553425 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.553346    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d50505-5262-4b95-968b-15aacb712df0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f1d50505-5262-4b95-968b-15aacb712df0" (UID: "f1d50505-5262-4b95-968b-15aacb712df0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:23:30.553425 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.553359    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1d50505-5262-4b95-968b-15aacb712df0-home\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:23:30.555296 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.555268    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d50505-5262-4b95-968b-15aacb712df0-dshm" (OuterVolumeSpecName: "dshm") pod "f1d50505-5262-4b95-968b-15aacb712df0" (UID: "f1d50505-5262-4b95-968b-15aacb712df0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:23:30.555296 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.555282    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1d50505-5262-4b95-968b-15aacb712df0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f1d50505-5262-4b95-968b-15aacb712df0" (UID: "f1d50505-5262-4b95-968b-15aacb712df0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:23:30.555449 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.555320    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d50505-5262-4b95-968b-15aacb712df0-kube-api-access-cxx8f" (OuterVolumeSpecName: "kube-api-access-cxx8f") pod "f1d50505-5262-4b95-968b-15aacb712df0" (UID: "f1d50505-5262-4b95-968b-15aacb712df0"). InnerVolumeSpecName "kube-api-access-cxx8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:23:30.599700 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.599662    2566 generic.go:356] "Generic (PLEG): container finished" podID="f1d50505-5262-4b95-968b-15aacb712df0" containerID="f96cf412976bf6df8b66d24e60a11a2fdc9d1169f83d28679d2abbcf841cf61c" exitCode=0
Jul 27 19:23:30.599885 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.599707    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"
Jul 27 19:23:30.599885 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.599750    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s" event={"ID":"f1d50505-5262-4b95-968b-15aacb712df0","Type":"ContainerDied","Data":"f96cf412976bf6df8b66d24e60a11a2fdc9d1169f83d28679d2abbcf841cf61c"}
Jul 27 19:23:30.599885 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.599793    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s" event={"ID":"f1d50505-5262-4b95-968b-15aacb712df0","Type":"ContainerDied","Data":"bd77e3d7ca0fb8067d98d794064096c448467e1035b9a8123d3239349492d35a"}
Jul 27 19:23:30.599885 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.599809    2566 scope.go:117] "RemoveContainer" containerID="f96cf412976bf6df8b66d24e60a11a2fdc9d1169f83d28679d2abbcf841cf61c"
Jul 27 19:23:30.601290 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.601264    2566 generic.go:356] "Generic (PLEG): container finished" podID="627605e0-2df0-40ee-a816-62ab2219ff69" containerID="be1555435433cd39c9a4f6a855175daec2b1a705e2ac2395ab30e114f4944a4d" exitCode=0
Jul 27 19:23:30.601388 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.601336    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7701-140320378424320" event={"ID":"627605e0-2df0-40ee-a816-62ab2219ff69","Type":"ContainerDied","Data":"be1555435433cd39c9a4f6a855175daec2b1a705e2ac2395ab30e114f4944a4d"}
Jul 27 19:23:30.609963 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.609945    2566 scope.go:117] "RemoveContainer" containerID="f96cf412976bf6df8b66d24e60a11a2fdc9d1169f83d28679d2abbcf841cf61c"
Jul 27 19:23:30.610276 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:23:30.610257    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f96cf412976bf6df8b66d24e60a11a2fdc9d1169f83d28679d2abbcf841cf61c\": container with ID starting with f96cf412976bf6df8b66d24e60a11a2fdc9d1169f83d28679d2abbcf841cf61c not found: ID does not exist" containerID="f96cf412976bf6df8b66d24e60a11a2fdc9d1169f83d28679d2abbcf841cf61c"
Jul 27 19:23:30.610336 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.610288    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f96cf412976bf6df8b66d24e60a11a2fdc9d1169f83d28679d2abbcf841cf61c"} err="failed to get container status \"f96cf412976bf6df8b66d24e60a11a2fdc9d1169f83d28679d2abbcf841cf61c\": rpc error: code = NotFound desc = could not find container \"f96cf412976bf6df8b66d24e60a11a2fdc9d1169f83d28679d2abbcf841cf61c\": container with ID starting with f96cf412976bf6df8b66d24e60a11a2fdc9d1169f83d28679d2abbcf841cf61c not found: ID does not exist"
Jul 27 19:23:30.638812 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.638782    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"]
Jul 27 19:23:30.643086 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.643061    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-7f7473/rate-test-model-7f7473-kserve-f5f7bf5f-92t6s"]
Jul 27 19:23:30.654156 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.654132    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxx8f\" (UniqueName: \"kubernetes.io/projected/f1d50505-5262-4b95-968b-15aacb712df0-kube-api-access-cxx8f\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:23:30.654156 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.654155    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1d50505-5262-4b95-968b-15aacb712df0-dshm\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:23:30.654301 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.654166    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1d50505-5262-4b95-968b-15aacb712df0-model-cache\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:23:30.654301 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.654174    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1d50505-5262-4b95-968b-15aacb712df0-tls-certs\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:23:30.654301 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:30.654181    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1d50505-5262-4b95-968b-15aacb712df0-tmp-dir\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:23:31.752649 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:31.752621    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-140320378424320"
Jul 27 19:23:31.783141 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:31.783094    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7701-140320378424320_627605e0-2df0-40ee-a816-62ab2219ff69/test-curl-7701-140320378424320/0.log"
Jul 27 19:23:31.814712 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:31.814675    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7701-140320378424320"]
Jul 27 19:23:31.817026 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:31.817005    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7701-140320378424320"]
Jul 27 19:23:31.867055 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:31.866965    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qqpm\" (UniqueName: \"kubernetes.io/projected/627605e0-2df0-40ee-a816-62ab2219ff69-kube-api-access-5qqpm\") pod \"627605e0-2df0-40ee-a816-62ab2219ff69\" (UID: \"627605e0-2df0-40ee-a816-62ab2219ff69\") "
Jul 27 19:23:31.869257 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:31.869234    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/627605e0-2df0-40ee-a816-62ab2219ff69-kube-api-access-5qqpm" (OuterVolumeSpecName: "kube-api-access-5qqpm") pod "627605e0-2df0-40ee-a816-62ab2219ff69" (UID: "627605e0-2df0-40ee-a816-62ab2219ff69"). InnerVolumeSpecName "kube-api-access-5qqpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:23:31.967965 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:31.967927    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qqpm\" (UniqueName: \"kubernetes.io/projected/627605e0-2df0-40ee-a816-62ab2219ff69-kube-api-access-5qqpm\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:23:32.124999 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:32.124921    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="627605e0-2df0-40ee-a816-62ab2219ff69" path="/var/lib/kubelet/pods/627605e0-2df0-40ee-a816-62ab2219ff69/volumes"
Jul 27 19:23:32.125331 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:32.125318    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1d50505-5262-4b95-968b-15aacb712df0" path="/var/lib/kubelet/pods/f1d50505-5262-4b95-968b-15aacb712df0/volumes"
Jul 27 19:23:32.615860 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:32.615830    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7701-140320378424320"
Jul 27 19:23:32.616029 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:32.615884    2566 scope.go:117] "RemoveContainer" containerID="be1555435433cd39c9a4f6a855175daec2b1a705e2ac2395ab30e114f4944a4d"
Jul 27 19:23:40.504192 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:40.504153    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc"]
Jul 27 19:23:40.504754 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:40.504735    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1d50505-5262-4b95-968b-15aacb712df0" containerName="main"
Jul 27 19:23:40.504849 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:40.504755    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1d50505-5262-4b95-968b-15aacb712df0" containerName="main"
Jul 27 19:23:40.504849 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:40.504776    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="627605e0-2df0-40ee-a816-62ab2219ff69" containerName="test-curl-7701-140320378424320"
Jul 27 19:23:40.504849 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:40.504785    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="627605e0-2df0-40ee-a816-62ab2219ff69" containerName="test-curl-7701-140320378424320"
Jul 27 19:23:40.505005 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:40.504876    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="627605e0-2df0-40ee-a816-62ab2219ff69" containerName="test-curl-7701-140320378424320"
Jul 27 19:23:40.505005 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:40.504894    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1d50505-5262-4b95-968b-15aacb712df0" containerName="main"
Jul 27 19:23:40.508042 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:40.508021    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc"
Jul 27 19:23:40.512113 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:40.512093    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-1d1107-serving-cert\""
Jul 27 19:23:40.529303 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:40.529272    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc"]
Jul 27 19:23:40.549279 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:40.549248    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsw4k\" (UniqueName: \"kubernetes.io/projected/e13dde1c-97a3-4284-ab61-e15c321893d1-kube-api-access-fsw4k\") pod \"maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc\" (UID: \"e13dde1c-97a3-4284-ab61-e15c321893d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc"
Jul 27 19:23:40.549412 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:40.549359    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13dde1c-97a3-4284-ab61-e15c321893d1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc\" (UID: \"e13dde1c-97a3-4284-ab61-e15c321893d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc"
Jul 27 19:23:40.549453 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:40.549407    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13dde1c-97a3-4284-ab61-e15c321893d1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc\" (UID: \"e13dde1c-97a3-4284-ab61-e15c321893d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc"
Jul 27 19:23:40.650284 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:40.650249    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13dde1c-97a3-4284-ab61-e15c321893d1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc\" (UID: \"e13dde1c-97a3-4284-ab61-e15c321893d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc"
Jul 27 19:23:40.650284 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:40.650299    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsw4k\" (UniqueName: \"kubernetes.io/projected/e13dde1c-97a3-4284-ab61-e15c321893d1-kube-api-access-fsw4k\") pod \"maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc\" (UID: \"e13dde1c-97a3-4284-ab61-e15c321893d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc"
Jul 27 19:23:40.650509 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:40.650388    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13dde1c-97a3-4284-ab61-e15c321893d1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc\" (UID: \"e13dde1c-97a3-4284-ab61-e15c321893d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc"
Jul 27 19:23:40.650509 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:23:40.650412    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-serving-cert: secret "maas-api-e2e-ipp-a-1d1107-serving-cert" not found
Jul 27 19:23:40.650509 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:23:40.650472    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e13dde1c-97a3-4284-ab61-e15c321893d1-maas-api-tls podName:e13dde1c-97a3-4284-ab61-e15c321893d1 nodeName:}" failed. No retries permitted until 2026-07-27 19:23:41.150454863 +0000 UTC m=+3281.584712452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e13dde1c-97a3-4284-ab61-e15c321893d1-maas-api-tls") pod "maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc" (UID: "e13dde1c-97a3-4284-ab61-e15c321893d1") : secret "maas-api-e2e-ipp-a-1d1107-serving-cert" not found
Jul 27 19:23:40.651011 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:40.650990    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13dde1c-97a3-4284-ab61-e15c321893d1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc\" (UID: \"e13dde1c-97a3-4284-ab61-e15c321893d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc"
Jul 27 19:23:40.663489 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:40.663459    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsw4k\" (UniqueName: \"kubernetes.io/projected/e13dde1c-97a3-4284-ab61-e15c321893d1-kube-api-access-fsw4k\") pod \"maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc\" (UID: \"e13dde1c-97a3-4284-ab61-e15c321893d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc"
Jul 27 19:23:41.155610 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:41.155568    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13dde1c-97a3-4284-ab61-e15c321893d1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc\" (UID: \"e13dde1c-97a3-4284-ab61-e15c321893d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc"
Jul 27 19:23:41.158341 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:41.158317    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13dde1c-97a3-4284-ab61-e15c321893d1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc\" (UID: \"e13dde1c-97a3-4284-ab61-e15c321893d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc"
Jul 27 19:23:41.418773 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:41.418685    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc"
Jul 27 19:23:41.578507 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:41.578480    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc"]
Jul 27 19:23:41.580823 ip-10-0-128-196 kubenswrapper[2566]: W0727 19:23:41.580794    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13dde1c_97a3_4284_ab61_e15c321893d1.slice/crio-7b563b1aeaf6826364ae50e749ace39f8eb103ad4aa85111e349db87ef0d8114 WatchSource:0}: Error finding container 7b563b1aeaf6826364ae50e749ace39f8eb103ad4aa85111e349db87ef0d8114: Status 404 returned error can't find the container with id 7b563b1aeaf6826364ae50e749ace39f8eb103ad4aa85111e349db87ef0d8114
Jul 27 19:23:41.667805 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:41.667768    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc" event={"ID":"e13dde1c-97a3-4284-ab61-e15c321893d1","Type":"ContainerStarted","Data":"7b563b1aeaf6826364ae50e749ace39f8eb103ad4aa85111e349db87ef0d8114"}
Jul 27 19:23:42.677430 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:42.677390    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc" event={"ID":"e13dde1c-97a3-4284-ab61-e15c321893d1","Type":"ContainerStarted","Data":"9ff47f93c5b65d25be4a8e9c9d20cd068790d89efc48bd66e9bbf0e0e3702c8d"}
Jul 27 19:23:42.677942 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:42.677466    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc"
Jul 27 19:23:42.700243 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:42.700193    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc" podStartSLOduration=2.093895516 podStartE2EDuration="2.700176318s" totalImagesPullingTime="606.280802ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:23:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:23:42.698723716 +0000 UTC m=+3283.132981327" watchObservedRunningTime="2026-07-27 19:23:42.700176318 +0000 UTC m=+3283.134433929"
Jul 27 19:23:48.688985 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:23:48.688954    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc"
Jul 27 19:24:00.178922 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:00.178890    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9"]
Jul 27 19:24:00.182393 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:00.182366    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9"
Jul 27 19:24:00.185338 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:00.185317    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-0bd532-serving-cert\""
Jul 27 19:24:00.199165 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:00.199137    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9"]
Jul 27 19:24:00.230569 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:00.230532    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6ngn\" (UniqueName: \"kubernetes.io/projected/7d5fef26-c31b-42b0-9015-06c78b08c5b3-kube-api-access-t6ngn\") pod \"maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9\" (UID: \"7d5fef26-c31b-42b0-9015-06c78b08c5b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9"
Jul 27 19:24:00.230739 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:00.230674    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d5fef26-c31b-42b0-9015-06c78b08c5b3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9\" (UID: \"7d5fef26-c31b-42b0-9015-06c78b08c5b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9"
Jul 27 19:24:00.230739 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:00.230715    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d5fef26-c31b-42b0-9015-06c78b08c5b3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9\" (UID: \"7d5fef26-c31b-42b0-9015-06c78b08c5b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9"
Jul 27 19:24:00.331816 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:00.331787    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d5fef26-c31b-42b0-9015-06c78b08c5b3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9\" (UID: \"7d5fef26-c31b-42b0-9015-06c78b08c5b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9"
Jul 27 19:24:00.332008 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:00.331825    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d5fef26-c31b-42b0-9015-06c78b08c5b3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9\" (UID: \"7d5fef26-c31b-42b0-9015-06c78b08c5b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9"
Jul 27 19:24:00.332008 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:00.331859    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6ngn\" (UniqueName: \"kubernetes.io/projected/7d5fef26-c31b-42b0-9015-06c78b08c5b3-kube-api-access-t6ngn\") pod \"maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9\" (UID: \"7d5fef26-c31b-42b0-9015-06c78b08c5b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9"
Jul 27 19:24:00.332578 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:00.332556    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d5fef26-c31b-42b0-9015-06c78b08c5b3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9\" (UID: \"7d5fef26-c31b-42b0-9015-06c78b08c5b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9"
Jul 27 19:24:00.334516 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:00.334491    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d5fef26-c31b-42b0-9015-06c78b08c5b3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9\" (UID: \"7d5fef26-c31b-42b0-9015-06c78b08c5b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9"
Jul 27 19:24:00.342567 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:00.342544    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6ngn\" (UniqueName: \"kubernetes.io/projected/7d5fef26-c31b-42b0-9015-06c78b08c5b3-kube-api-access-t6ngn\") pod \"maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9\" (UID: \"7d5fef26-c31b-42b0-9015-06c78b08c5b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9"
Jul 27 19:24:00.363436 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:00.363412    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 19:24:00.378995 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:00.378967    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 19:24:00.493364 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:00.493273    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9"
Jul 27 19:24:00.706496 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:00.706458    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9"]
Jul 27 19:24:00.713243 ip-10-0-128-196 kubenswrapper[2566]: W0727 19:24:00.713207    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5fef26_c31b_42b0_9015_06c78b08c5b3.slice/crio-129d80e455a89f6d524b2ba7ed251eca17674a5bfff41a73fcdf51b8e4d6a59a WatchSource:0}: Error finding container 129d80e455a89f6d524b2ba7ed251eca17674a5bfff41a73fcdf51b8e4d6a59a: Status 404 returned error can't find the container with id 129d80e455a89f6d524b2ba7ed251eca17674a5bfff41a73fcdf51b8e4d6a59a
Jul 27 19:24:00.794974 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:00.794937    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9" event={"ID":"7d5fef26-c31b-42b0-9015-06c78b08c5b3","Type":"ContainerStarted","Data":"129d80e455a89f6d524b2ba7ed251eca17674a5bfff41a73fcdf51b8e4d6a59a"}
Jul 27 19:24:01.801757 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:01.801717    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9" event={"ID":"7d5fef26-c31b-42b0-9015-06c78b08c5b3","Type":"ContainerStarted","Data":"d57bd0658502f944b5a2ff45029fa7200ff6e62a61e96d354e5c37c428aac5b1"}
Jul 27 19:24:01.802187 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:01.801780    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9"
Jul 27 19:24:01.828140 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:01.824912    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9" podStartSLOduration=1.4788030810000001 podStartE2EDuration="1.824889995s" totalImagesPullingTime="346.086914ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:24:01.822796746 +0000 UTC m=+3302.257054357" watchObservedRunningTime="2026-07-27 19:24:01.824889995 +0000 UTC m=+3302.259147607"
Jul 27 19:24:07.812623 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:07.812593    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9"
Jul 27 19:24:28.505555 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:28.505481    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"]
Jul 27 19:24:28.509333 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:28.509312    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:28.512594 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:28.512573    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-1d1107\"/\"openshift-service-ca.crt\""
Jul 27 19:24:28.512712 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:28.512591    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-1d1107\"/\"kube-root-ca.crt\""
Jul 27 19:24:28.513904 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:28.513885    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-1d1107\"/\"ipp-route-1d1107-kserve-self-signed-certs\""
Jul 27 19:24:28.514014 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:28.513978    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-1d1107\"/\"default-dockercfg-4dcct\""
Jul 27 19:24:28.525597 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:28.525575    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"]
Jul 27 19:24:28.587676 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:28.587647    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6635a8a-0116-40b1-b108-17e1aaad2f44-home\") pod \"ipp-route-1d1107-kserve-77667b57b7-znqmh\" (UID: \"e6635a8a-0116-40b1-b108-17e1aaad2f44\") " pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:28.587857 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:28.587713    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7snms\" (UniqueName: \"kubernetes.io/projected/e6635a8a-0116-40b1-b108-17e1aaad2f44-kube-api-access-7snms\") pod \"ipp-route-1d1107-kserve-77667b57b7-znqmh\" (UID: \"e6635a8a-0116-40b1-b108-17e1aaad2f44\") " pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:28.587857 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:28.587748    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6635a8a-0116-40b1-b108-17e1aaad2f44-model-cache\") pod \"ipp-route-1d1107-kserve-77667b57b7-znqmh\" (UID: \"e6635a8a-0116-40b1-b108-17e1aaad2f44\") " pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:28.587857 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:28.587790    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6635a8a-0116-40b1-b108-17e1aaad2f44-dshm\") pod \"ipp-route-1d1107-kserve-77667b57b7-znqmh\" (UID: \"e6635a8a-0116-40b1-b108-17e1aaad2f44\") " pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:28.587857 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:28.587821    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6635a8a-0116-40b1-b108-17e1aaad2f44-tmp-dir\") pod \"ipp-route-1d1107-kserve-77667b57b7-znqmh\" (UID: \"e6635a8a-0116-40b1-b108-17e1aaad2f44\") " pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:28.588035 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:28.587883    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6635a8a-0116-40b1-b108-17e1aaad2f44-tls-certs\") pod \"ipp-route-1d1107-kserve-77667b57b7-znqmh\" (UID: \"e6635a8a-0116-40b1-b108-17e1aaad2f44\") " pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:28.689195 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:28.689166    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6635a8a-0116-40b1-b108-17e1aaad2f44-home\") pod \"ipp-route-1d1107-kserve-77667b57b7-znqmh\" (UID: \"e6635a8a-0116-40b1-b108-17e1aaad2f44\") " pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:28.689358 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:28.689225    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7snms\" (UniqueName: \"kubernetes.io/projected/e6635a8a-0116-40b1-b108-17e1aaad2f44-kube-api-access-7snms\") pod \"ipp-route-1d1107-kserve-77667b57b7-znqmh\" (UID: \"e6635a8a-0116-40b1-b108-17e1aaad2f44\") " pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:28.689358 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:28.689261    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6635a8a-0116-40b1-b108-17e1aaad2f44-model-cache\") pod \"ipp-route-1d1107-kserve-77667b57b7-znqmh\" (UID: \"e6635a8a-0116-40b1-b108-17e1aaad2f44\") " pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:28.689358 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:28.689308    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6635a8a-0116-40b1-b108-17e1aaad2f44-dshm\") pod \"ipp-route-1d1107-kserve-77667b57b7-znqmh\" (UID: \"e6635a8a-0116-40b1-b108-17e1aaad2f44\") " pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:28.689358 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:28.689329    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6635a8a-0116-40b1-b108-17e1aaad2f44-tmp-dir\") pod \"ipp-route-1d1107-kserve-77667b57b7-znqmh\" (UID: \"e6635a8a-0116-40b1-b108-17e1aaad2f44\") " pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:28.689606 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:28.689371    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6635a8a-0116-40b1-b108-17e1aaad2f44-tls-certs\") pod \"ipp-route-1d1107-kserve-77667b57b7-znqmh\" (UID: \"e6635a8a-0116-40b1-b108-17e1aaad2f44\") " pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:28.689663 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:28.689604    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6635a8a-0116-40b1-b108-17e1aaad2f44-home\") pod \"ipp-route-1d1107-kserve-77667b57b7-znqmh\" (UID: \"e6635a8a-0116-40b1-b108-17e1aaad2f44\") " pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:28.689663 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:28.689650    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6635a8a-0116-40b1-b108-17e1aaad2f44-model-cache\") pod \"ipp-route-1d1107-kserve-77667b57b7-znqmh\" (UID: \"e6635a8a-0116-40b1-b108-17e1aaad2f44\") " pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:28.689776 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:28.689680    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6635a8a-0116-40b1-b108-17e1aaad2f44-tmp-dir\") pod \"ipp-route-1d1107-kserve-77667b57b7-znqmh\" (UID: \"e6635a8a-0116-40b1-b108-17e1aaad2f44\") " pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:28.691676 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:28.691656    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6635a8a-0116-40b1-b108-17e1aaad2f44-dshm\") pod \"ipp-route-1d1107-kserve-77667b57b7-znqmh\" (UID: \"e6635a8a-0116-40b1-b108-17e1aaad2f44\") " pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:28.691999 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:28.691982    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6635a8a-0116-40b1-b108-17e1aaad2f44-tls-certs\") pod \"ipp-route-1d1107-kserve-77667b57b7-znqmh\" (UID: \"e6635a8a-0116-40b1-b108-17e1aaad2f44\") " pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:28.698723 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:28.698699    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7snms\" (UniqueName: \"kubernetes.io/projected/e6635a8a-0116-40b1-b108-17e1aaad2f44-kube-api-access-7snms\") pod \"ipp-route-1d1107-kserve-77667b57b7-znqmh\" (UID: \"e6635a8a-0116-40b1-b108-17e1aaad2f44\") " pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:28.820307 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:28.820280    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:31.413588 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:24:31.413562    2566 log.go:32] "RunPodSandbox from runtime service failed" err=<
Jul 27 19:24:31.413588 ip-10-0-128-196 kubenswrapper[2566]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_ipp-route-1d1107-kserve-77667b57b7-znqmh_ai-tenant-e2e-ipp-a-1d1107_e6635a8a-0116-40b1-b108-17e1aaad2f44_0(da17ea025938dbfddae9c4a0d960296d6f99fe58b67a8551e491534370b93f0f): error adding pod ai-tenant-e2e-ipp-a-1d1107_ipp-route-1d1107-kserve-77667b57b7-znqmh to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"da17ea025938dbfddae9c4a0d960296d6f99fe58b67a8551e491534370b93f0f" Netns:"/var/run/netns/747f00d7-4650-433d-b265-aa0372839130" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-ipp-a-1d1107;K8S_POD_NAME=ipp-route-1d1107-kserve-77667b57b7-znqmh;K8S_POD_INFRA_CONTAINER_ID=da17ea025938dbfddae9c4a0d960296d6f99fe58b67a8551e491534370b93f0f;K8S_POD_UID=e6635a8a-0116-40b1-b108-17e1aaad2f44" Path:"" ERRORED: error configuring pod [ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh] networking: Multus: [ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh/e6635a8a-0116-40b1-b108-17e1aaad2f44]: error waiting for pod: Get "https://a21232e2c335645078a0372e432f61e3-d6350540945ea601.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-ipp-a-1d1107/pods/ipp-route-1d1107-kserve-77667b57b7-znqmh?timeout=1m0s": context deadline exceeded
Jul 27 19:24:31.413588 ip-10-0-128-196 kubenswrapper[2566]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 27 19:24:31.413588 ip-10-0-128-196 kubenswrapper[2566]:  >
Jul 27 19:24:31.414036 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:24:31.413614    2566 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Jul 27 19:24:31.414036 ip-10-0-128-196 kubenswrapper[2566]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_ipp-route-1d1107-kserve-77667b57b7-znqmh_ai-tenant-e2e-ipp-a-1d1107_e6635a8a-0116-40b1-b108-17e1aaad2f44_0(da17ea025938dbfddae9c4a0d960296d6f99fe58b67a8551e491534370b93f0f): error adding pod ai-tenant-e2e-ipp-a-1d1107_ipp-route-1d1107-kserve-77667b57b7-znqmh to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"da17ea025938dbfddae9c4a0d960296d6f99fe58b67a8551e491534370b93f0f" Netns:"/var/run/netns/747f00d7-4650-433d-b265-aa0372839130" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-ipp-a-1d1107;K8S_POD_NAME=ipp-route-1d1107-kserve-77667b57b7-znqmh;K8S_POD_INFRA_CONTAINER_ID=da17ea025938dbfddae9c4a0d960296d6f99fe58b67a8551e491534370b93f0f;K8S_POD_UID=e6635a8a-0116-40b1-b108-17e1aaad2f44" Path:"" ERRORED: error configuring pod [ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh] networking: Multus: [ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh/e6635a8a-0116-40b1-b108-17e1aaad2f44]: error waiting for pod: Get "https://a21232e2c335645078a0372e432f61e3-d6350540945ea601.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-ipp-a-1d1107/pods/ipp-route-1d1107-kserve-77667b57b7-znqmh?timeout=1m0s": context deadline exceeded
Jul 27 19:24:31.414036 ip-10-0-128-196 kubenswrapper[2566]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 27 19:24:31.414036 ip-10-0-128-196 kubenswrapper[2566]:  > pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:31.414036 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:24:31.413631    2566 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=<
Jul 27 19:24:31.414036 ip-10-0-128-196 kubenswrapper[2566]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_ipp-route-1d1107-kserve-77667b57b7-znqmh_ai-tenant-e2e-ipp-a-1d1107_e6635a8a-0116-40b1-b108-17e1aaad2f44_0(da17ea025938dbfddae9c4a0d960296d6f99fe58b67a8551e491534370b93f0f): error adding pod ai-tenant-e2e-ipp-a-1d1107_ipp-route-1d1107-kserve-77667b57b7-znqmh to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"da17ea025938dbfddae9c4a0d960296d6f99fe58b67a8551e491534370b93f0f" Netns:"/var/run/netns/747f00d7-4650-433d-b265-aa0372839130" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-ipp-a-1d1107;K8S_POD_NAME=ipp-route-1d1107-kserve-77667b57b7-znqmh;K8S_POD_INFRA_CONTAINER_ID=da17ea025938dbfddae9c4a0d960296d6f99fe58b67a8551e491534370b93f0f;K8S_POD_UID=e6635a8a-0116-40b1-b108-17e1aaad2f44" Path:"" ERRORED: error configuring pod [ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh] networking: Multus: [ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh/e6635a8a-0116-40b1-b108-17e1aaad2f44]: error waiting for pod: Get "https://a21232e2c335645078a0372e432f61e3-d6350540945ea601.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-ipp-a-1d1107/pods/ipp-route-1d1107-kserve-77667b57b7-znqmh?timeout=1m0s": context deadline exceeded
Jul 27 19:24:31.414036 ip-10-0-128-196 kubenswrapper[2566]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 27 19:24:31.414036 ip-10-0-128-196 kubenswrapper[2566]:  > pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:31.414316 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:24:31.413691    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"ipp-route-1d1107-kserve-77667b57b7-znqmh_ai-tenant-e2e-ipp-a-1d1107(e6635a8a-0116-40b1-b108-17e1aaad2f44)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"ipp-route-1d1107-kserve-77667b57b7-znqmh_ai-tenant-e2e-ipp-a-1d1107(e6635a8a-0116-40b1-b108-17e1aaad2f44)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_ipp-route-1d1107-kserve-77667b57b7-znqmh_ai-tenant-e2e-ipp-a-1d1107_e6635a8a-0116-40b1-b108-17e1aaad2f44_0(da17ea025938dbfddae9c4a0d960296d6f99fe58b67a8551e491534370b93f0f): error adding pod ai-tenant-e2e-ipp-a-1d1107_ipp-route-1d1107-kserve-77667b57b7-znqmh to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"da17ea025938dbfddae9c4a0d960296d6f99fe58b67a8551e491534370b93f0f\\\" Netns:\\\"/var/run/netns/747f00d7-4650-433d-b265-aa0372839130\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-ipp-a-1d1107;K8S_POD_NAME=ipp-route-1d1107-kserve-77667b57b7-znqmh;K8S_POD_INFRA_CONTAINER_ID=da17ea025938dbfddae9c4a0d960296d6f99fe58b67a8551e491534370b93f0f;K8S_POD_UID=e6635a8a-0116-40b1-b108-17e1aaad2f44\\\" Path:\\\"\\\" ERRORED: error configuring pod [ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh] networking: Multus: [ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh/e6635a8a-0116-40b1-b108-17e1aaad2f44]: error waiting for pod: Get \\\"https://a21232e2c335645078a0372e432f61e3-d6350540945ea601.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-ipp-a-1d1107/pods/ipp-route-1d1107-kserve-77667b57b7-znqmh?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh" podUID="e6635a8a-0116-40b1-b108-17e1aaad2f44"
Jul 27 19:24:31.984576 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:31.984540    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:31.985069 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:31.985046    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:34.577259 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:24:34.577224    2566 log.go:32] "RunPodSandbox from runtime service failed" err=<
Jul 27 19:24:34.577259 ip-10-0-128-196 kubenswrapper[2566]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_ipp-route-1d1107-kserve-77667b57b7-znqmh_ai-tenant-e2e-ipp-a-1d1107_e6635a8a-0116-40b1-b108-17e1aaad2f44_0(d843599434be377fb3a01a3f9778f01c5731d8b1ac1398172d20bbbdc872d556): error adding pod ai-tenant-e2e-ipp-a-1d1107_ipp-route-1d1107-kserve-77667b57b7-znqmh to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"d843599434be377fb3a01a3f9778f01c5731d8b1ac1398172d20bbbdc872d556" Netns:"/var/run/netns/6b23fd94-fc9b-4a97-affc-0f00c344c9ac" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-ipp-a-1d1107;K8S_POD_NAME=ipp-route-1d1107-kserve-77667b57b7-znqmh;K8S_POD_INFRA_CONTAINER_ID=d843599434be377fb3a01a3f9778f01c5731d8b1ac1398172d20bbbdc872d556;K8S_POD_UID=e6635a8a-0116-40b1-b108-17e1aaad2f44" Path:"" ERRORED: error configuring pod [ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh] networking: Multus: [ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh/e6635a8a-0116-40b1-b108-17e1aaad2f44]: error waiting for pod: Get "https://a21232e2c335645078a0372e432f61e3-d6350540945ea601.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-ipp-a-1d1107/pods/ipp-route-1d1107-kserve-77667b57b7-znqmh?timeout=1m0s": context deadline exceeded
Jul 27 19:24:34.577259 ip-10-0-128-196 kubenswrapper[2566]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 27 19:24:34.577259 ip-10-0-128-196 kubenswrapper[2566]:  >
Jul 27 19:24:34.577638 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:24:34.577281    2566 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Jul 27 19:24:34.577638 ip-10-0-128-196 kubenswrapper[2566]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_ipp-route-1d1107-kserve-77667b57b7-znqmh_ai-tenant-e2e-ipp-a-1d1107_e6635a8a-0116-40b1-b108-17e1aaad2f44_0(d843599434be377fb3a01a3f9778f01c5731d8b1ac1398172d20bbbdc872d556): error adding pod ai-tenant-e2e-ipp-a-1d1107_ipp-route-1d1107-kserve-77667b57b7-znqmh to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"d843599434be377fb3a01a3f9778f01c5731d8b1ac1398172d20bbbdc872d556" Netns:"/var/run/netns/6b23fd94-fc9b-4a97-affc-0f00c344c9ac" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-ipp-a-1d1107;K8S_POD_NAME=ipp-route-1d1107-kserve-77667b57b7-znqmh;K8S_POD_INFRA_CONTAINER_ID=d843599434be377fb3a01a3f9778f01c5731d8b1ac1398172d20bbbdc872d556;K8S_POD_UID=e6635a8a-0116-40b1-b108-17e1aaad2f44" Path:"" ERRORED: error configuring pod [ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh] networking: Multus: [ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh/e6635a8a-0116-40b1-b108-17e1aaad2f44]: error waiting for pod: Get "https://a21232e2c335645078a0372e432f61e3-d6350540945ea601.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-ipp-a-1d1107/pods/ipp-route-1d1107-kserve-77667b57b7-znqmh?timeout=1m0s": context deadline exceeded
Jul 27 19:24:34.577638 ip-10-0-128-196 kubenswrapper[2566]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 27 19:24:34.577638 ip-10-0-128-196 kubenswrapper[2566]:  > pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:34.577638 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:24:34.577301    2566 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=<
Jul 27 19:24:34.577638 ip-10-0-128-196 kubenswrapper[2566]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_ipp-route-1d1107-kserve-77667b57b7-znqmh_ai-tenant-e2e-ipp-a-1d1107_e6635a8a-0116-40b1-b108-17e1aaad2f44_0(d843599434be377fb3a01a3f9778f01c5731d8b1ac1398172d20bbbdc872d556): error adding pod ai-tenant-e2e-ipp-a-1d1107_ipp-route-1d1107-kserve-77667b57b7-znqmh to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"d843599434be377fb3a01a3f9778f01c5731d8b1ac1398172d20bbbdc872d556" Netns:"/var/run/netns/6b23fd94-fc9b-4a97-affc-0f00c344c9ac" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-ipp-a-1d1107;K8S_POD_NAME=ipp-route-1d1107-kserve-77667b57b7-znqmh;K8S_POD_INFRA_CONTAINER_ID=d843599434be377fb3a01a3f9778f01c5731d8b1ac1398172d20bbbdc872d556;K8S_POD_UID=e6635a8a-0116-40b1-b108-17e1aaad2f44" Path:"" ERRORED: error configuring pod [ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh] networking: Multus: [ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh/e6635a8a-0116-40b1-b108-17e1aaad2f44]: error waiting for pod: Get "https://a21232e2c335645078a0372e432f61e3-d6350540945ea601.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-ipp-a-1d1107/pods/ipp-route-1d1107-kserve-77667b57b7-znqmh?timeout=1m0s": context deadline exceeded
Jul 27 19:24:34.577638 ip-10-0-128-196 kubenswrapper[2566]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 27 19:24:34.577638 ip-10-0-128-196 kubenswrapper[2566]:  > pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:34.577869 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:24:34.577385    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"ipp-route-1d1107-kserve-77667b57b7-znqmh_ai-tenant-e2e-ipp-a-1d1107(e6635a8a-0116-40b1-b108-17e1aaad2f44)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"ipp-route-1d1107-kserve-77667b57b7-znqmh_ai-tenant-e2e-ipp-a-1d1107(e6635a8a-0116-40b1-b108-17e1aaad2f44)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_ipp-route-1d1107-kserve-77667b57b7-znqmh_ai-tenant-e2e-ipp-a-1d1107_e6635a8a-0116-40b1-b108-17e1aaad2f44_0(d843599434be377fb3a01a3f9778f01c5731d8b1ac1398172d20bbbdc872d556): error adding pod ai-tenant-e2e-ipp-a-1d1107_ipp-route-1d1107-kserve-77667b57b7-znqmh to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"d843599434be377fb3a01a3f9778f01c5731d8b1ac1398172d20bbbdc872d556\\\" Netns:\\\"/var/run/netns/6b23fd94-fc9b-4a97-affc-0f00c344c9ac\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-ipp-a-1d1107;K8S_POD_NAME=ipp-route-1d1107-kserve-77667b57b7-znqmh;K8S_POD_INFRA_CONTAINER_ID=d843599434be377fb3a01a3f9778f01c5731d8b1ac1398172d20bbbdc872d556;K8S_POD_UID=e6635a8a-0116-40b1-b108-17e1aaad2f44\\\" Path:\\\"\\\" ERRORED: error configuring pod [ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh] networking: Multus: [ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh/e6635a8a-0116-40b1-b108-17e1aaad2f44]: error waiting for pod: Get \\\"https://a21232e2c335645078a0372e432f61e3-d6350540945ea601.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-ipp-a-1d1107/pods/ipp-route-1d1107-kserve-77667b57b7-znqmh?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh" podUID="e6635a8a-0116-40b1-b108-17e1aaad2f44"
Jul 27 19:24:47.117668 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:47.117632    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:47.118261 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:24:47.118016    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:49.710809 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:24:49.710774    2566 log.go:32] "RunPodSandbox from runtime service failed" err=<
Jul 27 19:24:49.710809 ip-10-0-128-196 kubenswrapper[2566]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_ipp-route-1d1107-kserve-77667b57b7-znqmh_ai-tenant-e2e-ipp-a-1d1107_e6635a8a-0116-40b1-b108-17e1aaad2f44_0(2b97bc766d054e64d7ca42997314bab90a438f0e471ec0950f7183836034bab9): error adding pod ai-tenant-e2e-ipp-a-1d1107_ipp-route-1d1107-kserve-77667b57b7-znqmh to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"2b97bc766d054e64d7ca42997314bab90a438f0e471ec0950f7183836034bab9" Netns:"/var/run/netns/cf56389e-8165-4473-a06b-3f44fc6d43ab" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-ipp-a-1d1107;K8S_POD_NAME=ipp-route-1d1107-kserve-77667b57b7-znqmh;K8S_POD_INFRA_CONTAINER_ID=2b97bc766d054e64d7ca42997314bab90a438f0e471ec0950f7183836034bab9;K8S_POD_UID=e6635a8a-0116-40b1-b108-17e1aaad2f44" Path:"" ERRORED: error configuring pod [ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh] networking: Multus: [ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh/e6635a8a-0116-40b1-b108-17e1aaad2f44]: error waiting for pod: Get "https://a21232e2c335645078a0372e432f61e3-d6350540945ea601.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-ipp-a-1d1107/pods/ipp-route-1d1107-kserve-77667b57b7-znqmh?timeout=1m0s": context deadline exceeded
Jul 27 19:24:49.710809 ip-10-0-128-196 kubenswrapper[2566]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 27 19:24:49.710809 ip-10-0-128-196 kubenswrapper[2566]:  >
Jul 27 19:24:49.711282 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:24:49.710828    2566 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Jul 27 19:24:49.711282 ip-10-0-128-196 kubenswrapper[2566]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_ipp-route-1d1107-kserve-77667b57b7-znqmh_ai-tenant-e2e-ipp-a-1d1107_e6635a8a-0116-40b1-b108-17e1aaad2f44_0(2b97bc766d054e64d7ca42997314bab90a438f0e471ec0950f7183836034bab9): error adding pod ai-tenant-e2e-ipp-a-1d1107_ipp-route-1d1107-kserve-77667b57b7-znqmh to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"2b97bc766d054e64d7ca42997314bab90a438f0e471ec0950f7183836034bab9" Netns:"/var/run/netns/cf56389e-8165-4473-a06b-3f44fc6d43ab" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-ipp-a-1d1107;K8S_POD_NAME=ipp-route-1d1107-kserve-77667b57b7-znqmh;K8S_POD_INFRA_CONTAINER_ID=2b97bc766d054e64d7ca42997314bab90a438f0e471ec0950f7183836034bab9;K8S_POD_UID=e6635a8a-0116-40b1-b108-17e1aaad2f44" Path:"" ERRORED: error configuring pod [ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh] networking: Multus: [ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh/e6635a8a-0116-40b1-b108-17e1aaad2f44]: error waiting for pod: Get "https://a21232e2c335645078a0372e432f61e3-d6350540945ea601.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-ipp-a-1d1107/pods/ipp-route-1d1107-kserve-77667b57b7-znqmh?timeout=1m0s": context deadline exceeded
Jul 27 19:24:49.711282 ip-10-0-128-196 kubenswrapper[2566]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 27 19:24:49.711282 ip-10-0-128-196 kubenswrapper[2566]:  > pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:49.711282 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:24:49.710846    2566 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=<
Jul 27 19:24:49.711282 ip-10-0-128-196 kubenswrapper[2566]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_ipp-route-1d1107-kserve-77667b57b7-znqmh_ai-tenant-e2e-ipp-a-1d1107_e6635a8a-0116-40b1-b108-17e1aaad2f44_0(2b97bc766d054e64d7ca42997314bab90a438f0e471ec0950f7183836034bab9): error adding pod ai-tenant-e2e-ipp-a-1d1107_ipp-route-1d1107-kserve-77667b57b7-znqmh to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"2b97bc766d054e64d7ca42997314bab90a438f0e471ec0950f7183836034bab9" Netns:"/var/run/netns/cf56389e-8165-4473-a06b-3f44fc6d43ab" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-ipp-a-1d1107;K8S_POD_NAME=ipp-route-1d1107-kserve-77667b57b7-znqmh;K8S_POD_INFRA_CONTAINER_ID=2b97bc766d054e64d7ca42997314bab90a438f0e471ec0950f7183836034bab9;K8S_POD_UID=e6635a8a-0116-40b1-b108-17e1aaad2f44" Path:"" ERRORED: error configuring pod [ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh] networking: Multus: [ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh/e6635a8a-0116-40b1-b108-17e1aaad2f44]: error waiting for pod: Get "https://a21232e2c335645078a0372e432f61e3-d6350540945ea601.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-ipp-a-1d1107/pods/ipp-route-1d1107-kserve-77667b57b7-znqmh?timeout=1m0s": context deadline exceeded
Jul 27 19:24:49.711282 ip-10-0-128-196 kubenswrapper[2566]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 27 19:24:49.711282 ip-10-0-128-196 kubenswrapper[2566]:  > pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:24:49.711520 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:24:49.710921    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"ipp-route-1d1107-kserve-77667b57b7-znqmh_ai-tenant-e2e-ipp-a-1d1107(e6635a8a-0116-40b1-b108-17e1aaad2f44)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"ipp-route-1d1107-kserve-77667b57b7-znqmh_ai-tenant-e2e-ipp-a-1d1107(e6635a8a-0116-40b1-b108-17e1aaad2f44)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_ipp-route-1d1107-kserve-77667b57b7-znqmh_ai-tenant-e2e-ipp-a-1d1107_e6635a8a-0116-40b1-b108-17e1aaad2f44_0(2b97bc766d054e64d7ca42997314bab90a438f0e471ec0950f7183836034bab9): error adding pod ai-tenant-e2e-ipp-a-1d1107_ipp-route-1d1107-kserve-77667b57b7-znqmh to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"2b97bc766d054e64d7ca42997314bab90a438f0e471ec0950f7183836034bab9\\\" Netns:\\\"/var/run/netns/cf56389e-8165-4473-a06b-3f44fc6d43ab\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-ipp-a-1d1107;K8S_POD_NAME=ipp-route-1d1107-kserve-77667b57b7-znqmh;K8S_POD_INFRA_CONTAINER_ID=2b97bc766d054e64d7ca42997314bab90a438f0e471ec0950f7183836034bab9;K8S_POD_UID=e6635a8a-0116-40b1-b108-17e1aaad2f44\\\" Path:\\\"\\\" ERRORED: error configuring pod [ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh] networking: Multus: [ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh/e6635a8a-0116-40b1-b108-17e1aaad2f44]: error waiting for pod: Get \\\"https://a21232e2c335645078a0372e432f61e3-d6350540945ea601.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-ipp-a-1d1107/pods/ipp-route-1d1107-kserve-77667b57b7-znqmh?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh" podUID="e6635a8a-0116-40b1-b108-17e1aaad2f44"
Jul 27 19:25:00.124309 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:00.124276    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:25:00.124685 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:00.124555    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:25:00.269634 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:00.269608    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"]
Jul 27 19:25:00.270262 ip-10-0-128-196 kubenswrapper[2566]: W0727 19:25:00.270236    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6635a8a_0116_40b1_b108_17e1aaad2f44.slice/crio-25abac35b2c1f68476e7c4d5671cbc158335a6ebd1b53d33e07d0951ad6506ac WatchSource:0}: Error finding container 25abac35b2c1f68476e7c4d5671cbc158335a6ebd1b53d33e07d0951ad6506ac: Status 404 returned error can't find the container with id 25abac35b2c1f68476e7c4d5671cbc158335a6ebd1b53d33e07d0951ad6506ac
Jul 27 19:25:01.153177 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:01.153139    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh" event={"ID":"e6635a8a-0116-40b1-b108-17e1aaad2f44","Type":"ContainerStarted","Data":"0417dee7940e543421aa564bd672f5e068c49f86898b2d138c7184ce2187eef9"}
Jul 27 19:25:01.153177 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:01.153178    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh" event={"ID":"e6635a8a-0116-40b1-b108-17e1aaad2f44","Type":"ContainerStarted","Data":"25abac35b2c1f68476e7c4d5671cbc158335a6ebd1b53d33e07d0951ad6506ac"}
Jul 27 19:25:01.175838 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:01.175785    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh" podStartSLOduration=33.175770361 podStartE2EDuration="33.175770361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:25:01.173702979 +0000 UTC m=+3361.607960590" watchObservedRunningTime="2026-07-27 19:25:01.175770361 +0000 UTC m=+3361.610027972"
Jul 27 19:25:08.820694 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:08.820663    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:25:08.821081 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:08.820711    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:25:08.832943 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:08.832915    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:25:09.211957 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:09.211873    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:25:50.276540 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:50.276465    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9"]
Jul 27 19:25:50.287576 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:50.287534    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9"
Jul 27 19:25:50.301354 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:50.301331    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-fa6952-serving-cert\""
Jul 27 19:25:50.314564 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:50.314536    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9"]
Jul 27 19:25:50.441507 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:50.441473    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b1616d3-ff3a-48a6-b783-e486810a0545-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9\" (UID: \"8b1616d3-ff3a-48a6-b783-e486810a0545\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9"
Jul 27 19:25:50.441689 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:50.441529    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b1616d3-ff3a-48a6-b783-e486810a0545-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9\" (UID: \"8b1616d3-ff3a-48a6-b783-e486810a0545\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9"
Jul 27 19:25:50.441689 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:50.441600    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4dpg\" (UniqueName: \"kubernetes.io/projected/8b1616d3-ff3a-48a6-b783-e486810a0545-kube-api-access-z4dpg\") pod \"maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9\" (UID: \"8b1616d3-ff3a-48a6-b783-e486810a0545\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9"
Jul 27 19:25:50.542791 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:50.542696    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4dpg\" (UniqueName: \"kubernetes.io/projected/8b1616d3-ff3a-48a6-b783-e486810a0545-kube-api-access-z4dpg\") pod \"maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9\" (UID: \"8b1616d3-ff3a-48a6-b783-e486810a0545\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9"
Jul 27 19:25:50.542791 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:50.542767    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b1616d3-ff3a-48a6-b783-e486810a0545-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9\" (UID: \"8b1616d3-ff3a-48a6-b783-e486810a0545\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9"
Jul 27 19:25:50.542985 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:50.542831    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b1616d3-ff3a-48a6-b783-e486810a0545-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9\" (UID: \"8b1616d3-ff3a-48a6-b783-e486810a0545\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9"
Jul 27 19:25:50.542985 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:25:50.542873    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-serving-cert: secret "maas-api-e2e-ipp-cleanup-fa6952-serving-cert" not found
Jul 27 19:25:50.542985 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:25:50.542956    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b1616d3-ff3a-48a6-b783-e486810a0545-maas-api-tls podName:8b1616d3-ff3a-48a6-b783-e486810a0545 nodeName:}" failed. No retries permitted until 2026-07-27 19:25:51.042935757 +0000 UTC m=+3411.477193363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8b1616d3-ff3a-48a6-b783-e486810a0545-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9" (UID: "8b1616d3-ff3a-48a6-b783-e486810a0545") : secret "maas-api-e2e-ipp-cleanup-fa6952-serving-cert" not found
Jul 27 19:25:50.543504 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:50.543484    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b1616d3-ff3a-48a6-b783-e486810a0545-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9\" (UID: \"8b1616d3-ff3a-48a6-b783-e486810a0545\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9"
Jul 27 19:25:50.554345 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:50.554313    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4dpg\" (UniqueName: \"kubernetes.io/projected/8b1616d3-ff3a-48a6-b783-e486810a0545-kube-api-access-z4dpg\") pod \"maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9\" (UID: \"8b1616d3-ff3a-48a6-b783-e486810a0545\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9"
Jul 27 19:25:51.046446 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:51.046412    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b1616d3-ff3a-48a6-b783-e486810a0545-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9\" (UID: \"8b1616d3-ff3a-48a6-b783-e486810a0545\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9"
Jul 27 19:25:51.049217 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:51.049182    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b1616d3-ff3a-48a6-b783-e486810a0545-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9\" (UID: \"8b1616d3-ff3a-48a6-b783-e486810a0545\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9"
Jul 27 19:25:51.198602 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:51.198559    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9"
Jul 27 19:25:51.355235 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:51.355210    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9"]
Jul 27 19:25:51.357964 ip-10-0-128-196 kubenswrapper[2566]: W0727 19:25:51.357930    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1616d3_ff3a_48a6_b783_e486810a0545.slice/crio-fcf092543abed8ee6dd1c460df81e3276e714cbc20bf1e094b9fffc0ba1e8f9c WatchSource:0}: Error finding container fcf092543abed8ee6dd1c460df81e3276e714cbc20bf1e094b9fffc0ba1e8f9c: Status 404 returned error can't find the container with id fcf092543abed8ee6dd1c460df81e3276e714cbc20bf1e094b9fffc0ba1e8f9c
Jul 27 19:25:51.443999 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:51.443965    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9" event={"ID":"8b1616d3-ff3a-48a6-b783-e486810a0545","Type":"ContainerStarted","Data":"fcf092543abed8ee6dd1c460df81e3276e714cbc20bf1e094b9fffc0ba1e8f9c"}
Jul 27 19:25:52.451191 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:52.451149    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9" event={"ID":"8b1616d3-ff3a-48a6-b783-e486810a0545","Type":"ContainerStarted","Data":"7df383eda9fc7f453a9860e70be7e74880709fa7d93a2af68df74216e1374327"}
Jul 27 19:25:52.451593 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:52.451212    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9"
Jul 27 19:25:52.475948 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:52.475901    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9" podStartSLOduration=2.096522091 podStartE2EDuration="2.47588438s" totalImagesPullingTime="379.362289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:25:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:25:52.473259351 +0000 UTC m=+3412.907516974" watchObservedRunningTime="2026-07-27 19:25:52.47588438 +0000 UTC m=+3412.910141991"
Jul 27 19:25:58.463021 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:25:58.462990    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9"
Jul 27 19:26:00.677330 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:00.677296    2566 scope.go:117] "RemoveContainer" containerID="b651172ca4910777fb222252cf8919f81e2951eb06efd678cbff92325d9cd397"
Jul 27 19:26:15.881213 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:15.881173    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9"]
Jul 27 19:26:15.881703 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:15.881368    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9" podUID="8b1616d3-ff3a-48a6-b783-e486810a0545" containerName="maas-api" containerID="cri-o://7df383eda9fc7f453a9860e70be7e74880709fa7d93a2af68df74216e1374327" gracePeriod=30
Jul 27 19:26:16.135164 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:16.135085    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9"
Jul 27 19:26:16.189372 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:16.189334    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b1616d3-ff3a-48a6-b783-e486810a0545-openshift-service-ca\") pod \"8b1616d3-ff3a-48a6-b783-e486810a0545\" (UID: \"8b1616d3-ff3a-48a6-b783-e486810a0545\") "
Jul 27 19:26:16.189550 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:16.189436    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4dpg\" (UniqueName: \"kubernetes.io/projected/8b1616d3-ff3a-48a6-b783-e486810a0545-kube-api-access-z4dpg\") pod \"8b1616d3-ff3a-48a6-b783-e486810a0545\" (UID: \"8b1616d3-ff3a-48a6-b783-e486810a0545\") "
Jul 27 19:26:16.189550 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:16.189500    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b1616d3-ff3a-48a6-b783-e486810a0545-maas-api-tls\") pod \"8b1616d3-ff3a-48a6-b783-e486810a0545\" (UID: \"8b1616d3-ff3a-48a6-b783-e486810a0545\") "
Jul 27 19:26:16.189667 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:16.189651    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b1616d3-ff3a-48a6-b783-e486810a0545-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8b1616d3-ff3a-48a6-b783-e486810a0545" (UID: "8b1616d3-ff3a-48a6-b783-e486810a0545"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:26:16.189797 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:16.189782    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b1616d3-ff3a-48a6-b783-e486810a0545-openshift-service-ca\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:26:16.191699 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:16.191676    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b1616d3-ff3a-48a6-b783-e486810a0545-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8b1616d3-ff3a-48a6-b783-e486810a0545" (UID: "8b1616d3-ff3a-48a6-b783-e486810a0545"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:26:16.191699 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:16.191689    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1616d3-ff3a-48a6-b783-e486810a0545-kube-api-access-z4dpg" (OuterVolumeSpecName: "kube-api-access-z4dpg") pod "8b1616d3-ff3a-48a6-b783-e486810a0545" (UID: "8b1616d3-ff3a-48a6-b783-e486810a0545"). InnerVolumeSpecName "kube-api-access-z4dpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:26:16.291031 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:16.290995    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4dpg\" (UniqueName: \"kubernetes.io/projected/8b1616d3-ff3a-48a6-b783-e486810a0545-kube-api-access-z4dpg\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:26:16.291031 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:16.291024    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b1616d3-ff3a-48a6-b783-e486810a0545-maas-api-tls\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:26:16.599312 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:16.599277    2566 generic.go:356] "Generic (PLEG): container finished" podID="8b1616d3-ff3a-48a6-b783-e486810a0545" containerID="7df383eda9fc7f453a9860e70be7e74880709fa7d93a2af68df74216e1374327" exitCode=0
Jul 27 19:26:16.599488 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:16.599327    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9"
Jul 27 19:26:16.599488 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:16.599367    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9" event={"ID":"8b1616d3-ff3a-48a6-b783-e486810a0545","Type":"ContainerDied","Data":"7df383eda9fc7f453a9860e70be7e74880709fa7d93a2af68df74216e1374327"}
Jul 27 19:26:16.599488 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:16.599438    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9" event={"ID":"8b1616d3-ff3a-48a6-b783-e486810a0545","Type":"ContainerDied","Data":"fcf092543abed8ee6dd1c460df81e3276e714cbc20bf1e094b9fffc0ba1e8f9c"}
Jul 27 19:26:16.599488 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:16.599461    2566 scope.go:117] "RemoveContainer" containerID="7df383eda9fc7f453a9860e70be7e74880709fa7d93a2af68df74216e1374327"
Jul 27 19:26:16.609669 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:16.609646    2566 scope.go:117] "RemoveContainer" containerID="7df383eda9fc7f453a9860e70be7e74880709fa7d93a2af68df74216e1374327"
Jul 27 19:26:16.609931 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:26:16.609905    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7df383eda9fc7f453a9860e70be7e74880709fa7d93a2af68df74216e1374327\": container with ID starting with 7df383eda9fc7f453a9860e70be7e74880709fa7d93a2af68df74216e1374327 not found: ID does not exist" containerID="7df383eda9fc7f453a9860e70be7e74880709fa7d93a2af68df74216e1374327"
Jul 27 19:26:16.609969 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:16.609942    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7df383eda9fc7f453a9860e70be7e74880709fa7d93a2af68df74216e1374327"} err="failed to get container status \"7df383eda9fc7f453a9860e70be7e74880709fa7d93a2af68df74216e1374327\": rpc error: code = NotFound desc = could not find container \"7df383eda9fc7f453a9860e70be7e74880709fa7d93a2af68df74216e1374327\": container with ID starting with 7df383eda9fc7f453a9860e70be7e74880709fa7d93a2af68df74216e1374327 not found: ID does not exist"
Jul 27 19:26:16.626825 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:16.626802    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9"]
Jul 27 19:26:16.635338 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:16.635316    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fa6952-765b9bf4d4-pmrs9"]
Jul 27 19:26:18.124246 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:18.124217    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b1616d3-ff3a-48a6-b783-e486810a0545" path="/var/lib/kubelet/pods/8b1616d3-ff3a-48a6-b783-e486810a0545/volumes"
Jul 27 19:26:52.314169 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:52.314135    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc"]
Jul 27 19:26:52.314694 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:52.314339    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc" podUID="e13dde1c-97a3-4284-ab61-e15c321893d1" containerName="maas-api" containerID="cri-o://9ff47f93c5b65d25be4a8e9c9d20cd068790d89efc48bd66e9bbf0e0e3702c8d" gracePeriod=30
Jul 27 19:26:52.580669 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:52.580640    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc"
Jul 27 19:26:52.632732 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:52.632702    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsw4k\" (UniqueName: \"kubernetes.io/projected/e13dde1c-97a3-4284-ab61-e15c321893d1-kube-api-access-fsw4k\") pod \"e13dde1c-97a3-4284-ab61-e15c321893d1\" (UID: \"e13dde1c-97a3-4284-ab61-e15c321893d1\") "
Jul 27 19:26:52.632907 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:52.632754    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13dde1c-97a3-4284-ab61-e15c321893d1-maas-api-tls\") pod \"e13dde1c-97a3-4284-ab61-e15c321893d1\" (UID: \"e13dde1c-97a3-4284-ab61-e15c321893d1\") "
Jul 27 19:26:52.632907 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:52.632781    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13dde1c-97a3-4284-ab61-e15c321893d1-openshift-service-ca\") pod \"e13dde1c-97a3-4284-ab61-e15c321893d1\" (UID: \"e13dde1c-97a3-4284-ab61-e15c321893d1\") "
Jul 27 19:26:52.633256 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:52.633226    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13dde1c-97a3-4284-ab61-e15c321893d1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e13dde1c-97a3-4284-ab61-e15c321893d1" (UID: "e13dde1c-97a3-4284-ab61-e15c321893d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:26:52.634895 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:52.634871    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13dde1c-97a3-4284-ab61-e15c321893d1-kube-api-access-fsw4k" (OuterVolumeSpecName: "kube-api-access-fsw4k") pod "e13dde1c-97a3-4284-ab61-e15c321893d1" (UID: "e13dde1c-97a3-4284-ab61-e15c321893d1"). InnerVolumeSpecName "kube-api-access-fsw4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:26:52.634995 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:52.634946    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e13dde1c-97a3-4284-ab61-e15c321893d1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e13dde1c-97a3-4284-ab61-e15c321893d1" (UID: "e13dde1c-97a3-4284-ab61-e15c321893d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:26:52.733960 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:52.733932    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsw4k\" (UniqueName: \"kubernetes.io/projected/e13dde1c-97a3-4284-ab61-e15c321893d1-kube-api-access-fsw4k\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:26:52.733960 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:52.733957    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13dde1c-97a3-4284-ab61-e15c321893d1-maas-api-tls\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:26:52.734160 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:52.733967    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13dde1c-97a3-4284-ab61-e15c321893d1-openshift-service-ca\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:26:52.806464 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:52.806431    2566 generic.go:356] "Generic (PLEG): container finished" podID="e13dde1c-97a3-4284-ab61-e15c321893d1" containerID="9ff47f93c5b65d25be4a8e9c9d20cd068790d89efc48bd66e9bbf0e0e3702c8d" exitCode=0
Jul 27 19:26:52.806658 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:52.806481    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc"
Jul 27 19:26:52.806658 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:52.806515    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc" event={"ID":"e13dde1c-97a3-4284-ab61-e15c321893d1","Type":"ContainerDied","Data":"9ff47f93c5b65d25be4a8e9c9d20cd068790d89efc48bd66e9bbf0e0e3702c8d"}
Jul 27 19:26:52.806658 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:52.806563    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc" event={"ID":"e13dde1c-97a3-4284-ab61-e15c321893d1","Type":"ContainerDied","Data":"7b563b1aeaf6826364ae50e749ace39f8eb103ad4aa85111e349db87ef0d8114"}
Jul 27 19:26:52.806658 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:52.806583    2566 scope.go:117] "RemoveContainer" containerID="9ff47f93c5b65d25be4a8e9c9d20cd068790d89efc48bd66e9bbf0e0e3702c8d"
Jul 27 19:26:52.816755 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:52.816733    2566 scope.go:117] "RemoveContainer" containerID="9ff47f93c5b65d25be4a8e9c9d20cd068790d89efc48bd66e9bbf0e0e3702c8d"
Jul 27 19:26:52.817042 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:26:52.817024    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ff47f93c5b65d25be4a8e9c9d20cd068790d89efc48bd66e9bbf0e0e3702c8d\": container with ID starting with 9ff47f93c5b65d25be4a8e9c9d20cd068790d89efc48bd66e9bbf0e0e3702c8d not found: ID does not exist" containerID="9ff47f93c5b65d25be4a8e9c9d20cd068790d89efc48bd66e9bbf0e0e3702c8d"
Jul 27 19:26:52.817092 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:52.817052    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ff47f93c5b65d25be4a8e9c9d20cd068790d89efc48bd66e9bbf0e0e3702c8d"} err="failed to get container status \"9ff47f93c5b65d25be4a8e9c9d20cd068790d89efc48bd66e9bbf0e0e3702c8d\": rpc error: code = NotFound desc = could not find container \"9ff47f93c5b65d25be4a8e9c9d20cd068790d89efc48bd66e9bbf0e0e3702c8d\": container with ID starting with 9ff47f93c5b65d25be4a8e9c9d20cd068790d89efc48bd66e9bbf0e0e3702c8d not found: ID does not exist"
Jul 27 19:26:52.836523 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:52.836493    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc"]
Jul 27 19:26:52.848621 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:52.848595    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1d1107-754dc8b4fb-tb8hc"]
Jul 27 19:26:54.124386 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:54.124357    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e13dde1c-97a3-4284-ab61-e15c321893d1" path="/var/lib/kubelet/pods/e13dde1c-97a3-4284-ab61-e15c321893d1/volumes"
Jul 27 19:26:58.024059 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:26:58.024020    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"]
Jul 27 19:27:13.044248 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.044199    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh" podUID="e6635a8a-0116-40b1-b108-17e1aaad2f44" containerName="main" containerID="cri-o://0417dee7940e543421aa564bd672f5e068c49f86898b2d138c7184ce2187eef9" gracePeriod=45
Jul 27 19:27:13.299582 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.299526    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:27:13.425316 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.425286    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6635a8a-0116-40b1-b108-17e1aaad2f44-dshm\") pod \"e6635a8a-0116-40b1-b108-17e1aaad2f44\" (UID: \"e6635a8a-0116-40b1-b108-17e1aaad2f44\") "
Jul 27 19:27:13.425316 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.425330    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6635a8a-0116-40b1-b108-17e1aaad2f44-tls-certs\") pod \"e6635a8a-0116-40b1-b108-17e1aaad2f44\" (UID: \"e6635a8a-0116-40b1-b108-17e1aaad2f44\") "
Jul 27 19:27:13.425530 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.425352    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7snms\" (UniqueName: \"kubernetes.io/projected/e6635a8a-0116-40b1-b108-17e1aaad2f44-kube-api-access-7snms\") pod \"e6635a8a-0116-40b1-b108-17e1aaad2f44\" (UID: \"e6635a8a-0116-40b1-b108-17e1aaad2f44\") "
Jul 27 19:27:13.425530 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.425397    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6635a8a-0116-40b1-b108-17e1aaad2f44-home\") pod \"e6635a8a-0116-40b1-b108-17e1aaad2f44\" (UID: \"e6635a8a-0116-40b1-b108-17e1aaad2f44\") "
Jul 27 19:27:13.425530 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.425489    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6635a8a-0116-40b1-b108-17e1aaad2f44-tmp-dir\") pod \"e6635a8a-0116-40b1-b108-17e1aaad2f44\" (UID: \"e6635a8a-0116-40b1-b108-17e1aaad2f44\") "
Jul 27 19:27:13.425530 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.425514    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6635a8a-0116-40b1-b108-17e1aaad2f44-model-cache\") pod \"e6635a8a-0116-40b1-b108-17e1aaad2f44\" (UID: \"e6635a8a-0116-40b1-b108-17e1aaad2f44\") "
Jul 27 19:27:13.425750 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.425721    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6635a8a-0116-40b1-b108-17e1aaad2f44-home" (OuterVolumeSpecName: "home") pod "e6635a8a-0116-40b1-b108-17e1aaad2f44" (UID: "e6635a8a-0116-40b1-b108-17e1aaad2f44"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:27:13.425802 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.425734    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6635a8a-0116-40b1-b108-17e1aaad2f44-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e6635a8a-0116-40b1-b108-17e1aaad2f44" (UID: "e6635a8a-0116-40b1-b108-17e1aaad2f44"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:27:13.425851 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.425799    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6635a8a-0116-40b1-b108-17e1aaad2f44-model-cache" (OuterVolumeSpecName: "model-cache") pod "e6635a8a-0116-40b1-b108-17e1aaad2f44" (UID: "e6635a8a-0116-40b1-b108-17e1aaad2f44"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:27:13.425851 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.425817    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6635a8a-0116-40b1-b108-17e1aaad2f44-home\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:27:13.427565 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.427545    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6635a8a-0116-40b1-b108-17e1aaad2f44-dshm" (OuterVolumeSpecName: "dshm") pod "e6635a8a-0116-40b1-b108-17e1aaad2f44" (UID: "e6635a8a-0116-40b1-b108-17e1aaad2f44"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:27:13.427970 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.427946    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6635a8a-0116-40b1-b108-17e1aaad2f44-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6635a8a-0116-40b1-b108-17e1aaad2f44" (UID: "e6635a8a-0116-40b1-b108-17e1aaad2f44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:27:13.428018 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.427954    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6635a8a-0116-40b1-b108-17e1aaad2f44-kube-api-access-7snms" (OuterVolumeSpecName: "kube-api-access-7snms") pod "e6635a8a-0116-40b1-b108-17e1aaad2f44" (UID: "e6635a8a-0116-40b1-b108-17e1aaad2f44"). InnerVolumeSpecName "kube-api-access-7snms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:27:13.526489 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.526443    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6635a8a-0116-40b1-b108-17e1aaad2f44-tmp-dir\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:27:13.526489 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.526483    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6635a8a-0116-40b1-b108-17e1aaad2f44-model-cache\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:27:13.526489 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.526493    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6635a8a-0116-40b1-b108-17e1aaad2f44-dshm\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:27:13.526489 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.526501    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6635a8a-0116-40b1-b108-17e1aaad2f44-tls-certs\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:27:13.526751 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.526511    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7snms\" (UniqueName: \"kubernetes.io/projected/e6635a8a-0116-40b1-b108-17e1aaad2f44-kube-api-access-7snms\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:27:13.931428 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.931392    2566 generic.go:356] "Generic (PLEG): container finished" podID="e6635a8a-0116-40b1-b108-17e1aaad2f44" containerID="0417dee7940e543421aa564bd672f5e068c49f86898b2d138c7184ce2187eef9" exitCode=0
Jul 27 19:27:13.931631 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.931437    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"
Jul 27 19:27:13.931631 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.931467    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh" event={"ID":"e6635a8a-0116-40b1-b108-17e1aaad2f44","Type":"ContainerDied","Data":"0417dee7940e543421aa564bd672f5e068c49f86898b2d138c7184ce2187eef9"}
Jul 27 19:27:13.931631 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.931498    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh" event={"ID":"e6635a8a-0116-40b1-b108-17e1aaad2f44","Type":"ContainerDied","Data":"25abac35b2c1f68476e7c4d5671cbc158335a6ebd1b53d33e07d0951ad6506ac"}
Jul 27 19:27:13.931631 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.931515    2566 scope.go:117] "RemoveContainer" containerID="0417dee7940e543421aa564bd672f5e068c49f86898b2d138c7184ce2187eef9"
Jul 27 19:27:13.941246 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.941228    2566 scope.go:117] "RemoveContainer" containerID="0417dee7940e543421aa564bd672f5e068c49f86898b2d138c7184ce2187eef9"
Jul 27 19:27:13.941513 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:27:13.941495    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0417dee7940e543421aa564bd672f5e068c49f86898b2d138c7184ce2187eef9\": container with ID starting with 0417dee7940e543421aa564bd672f5e068c49f86898b2d138c7184ce2187eef9 not found: ID does not exist" containerID="0417dee7940e543421aa564bd672f5e068c49f86898b2d138c7184ce2187eef9"
Jul 27 19:27:13.941561 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.941525    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0417dee7940e543421aa564bd672f5e068c49f86898b2d138c7184ce2187eef9"} err="failed to get container status \"0417dee7940e543421aa564bd672f5e068c49f86898b2d138c7184ce2187eef9\": rpc error: code = NotFound desc = could not find container \"0417dee7940e543421aa564bd672f5e068c49f86898b2d138c7184ce2187eef9\": container with ID starting with 0417dee7940e543421aa564bd672f5e068c49f86898b2d138c7184ce2187eef9 not found: ID does not exist"
Jul 27 19:27:13.954736 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.954715    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"]
Jul 27 19:27:13.961204 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:13.961184    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-1d1107/ipp-route-1d1107-kserve-77667b57b7-znqmh"]
Jul 27 19:27:14.124273 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:14.124240    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6635a8a-0116-40b1-b108-17e1aaad2f44" path="/var/lib/kubelet/pods/e6635a8a-0116-40b1-b108-17e1aaad2f44/volumes"
Jul 27 19:27:55.999625 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:55.999587    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9"]
Jul 27 19:27:56.000095 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:55.999863    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9" podUID="7d5fef26-c31b-42b0-9015-06c78b08c5b3" containerName="maas-api" containerID="cri-o://d57bd0658502f944b5a2ff45029fa7200ff6e62a61e96d354e5c37c428aac5b1" gracePeriod=30
Jul 27 19:27:56.169988 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:56.169927    2566 generic.go:356] "Generic (PLEG): container finished" podID="7d5fef26-c31b-42b0-9015-06c78b08c5b3" containerID="d57bd0658502f944b5a2ff45029fa7200ff6e62a61e96d354e5c37c428aac5b1" exitCode=0
Jul 27 19:27:56.170084 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:56.170011    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9" event={"ID":"7d5fef26-c31b-42b0-9015-06c78b08c5b3","Type":"ContainerDied","Data":"d57bd0658502f944b5a2ff45029fa7200ff6e62a61e96d354e5c37c428aac5b1"}
Jul 27 19:27:56.247205 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:56.247182    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9"
Jul 27 19:27:56.327860 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:56.327829    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6ngn\" (UniqueName: \"kubernetes.io/projected/7d5fef26-c31b-42b0-9015-06c78b08c5b3-kube-api-access-t6ngn\") pod \"7d5fef26-c31b-42b0-9015-06c78b08c5b3\" (UID: \"7d5fef26-c31b-42b0-9015-06c78b08c5b3\") "
Jul 27 19:27:56.328039 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:56.327901    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d5fef26-c31b-42b0-9015-06c78b08c5b3-maas-api-tls\") pod \"7d5fef26-c31b-42b0-9015-06c78b08c5b3\" (UID: \"7d5fef26-c31b-42b0-9015-06c78b08c5b3\") "
Jul 27 19:27:56.328039 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:56.327959    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d5fef26-c31b-42b0-9015-06c78b08c5b3-openshift-service-ca\") pod \"7d5fef26-c31b-42b0-9015-06c78b08c5b3\" (UID: \"7d5fef26-c31b-42b0-9015-06c78b08c5b3\") "
Jul 27 19:27:56.328323 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:56.328292    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d5fef26-c31b-42b0-9015-06c78b08c5b3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7d5fef26-c31b-42b0-9015-06c78b08c5b3" (UID: "7d5fef26-c31b-42b0-9015-06c78b08c5b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:27:56.330160 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:56.330141    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5fef26-c31b-42b0-9015-06c78b08c5b3-kube-api-access-t6ngn" (OuterVolumeSpecName: "kube-api-access-t6ngn") pod "7d5fef26-c31b-42b0-9015-06c78b08c5b3" (UID: "7d5fef26-c31b-42b0-9015-06c78b08c5b3"). InnerVolumeSpecName "kube-api-access-t6ngn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:27:56.330236 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:56.330161    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d5fef26-c31b-42b0-9015-06c78b08c5b3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7d5fef26-c31b-42b0-9015-06c78b08c5b3" (UID: "7d5fef26-c31b-42b0-9015-06c78b08c5b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:27:56.429428 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:56.429395    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d5fef26-c31b-42b0-9015-06c78b08c5b3-openshift-service-ca\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:27:56.429428 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:56.429421    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6ngn\" (UniqueName: \"kubernetes.io/projected/7d5fef26-c31b-42b0-9015-06c78b08c5b3-kube-api-access-t6ngn\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:27:56.429428 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:56.429431    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d5fef26-c31b-42b0-9015-06c78b08c5b3-maas-api-tls\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:27:57.177571 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:57.177545    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9"
Jul 27 19:27:57.178019 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:57.177585    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9" event={"ID":"7d5fef26-c31b-42b0-9015-06c78b08c5b3","Type":"ContainerDied","Data":"129d80e455a89f6d524b2ba7ed251eca17674a5bfff41a73fcdf51b8e4d6a59a"}
Jul 27 19:27:57.178019 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:57.177652    2566 scope.go:117] "RemoveContainer" containerID="d57bd0658502f944b5a2ff45029fa7200ff6e62a61e96d354e5c37c428aac5b1"
Jul 27 19:27:57.200472 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:57.200449    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9"]
Jul 27 19:27:57.206602 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:57.206580    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0bd532-67bcdccb55-97nm9"]
Jul 27 19:27:58.124505 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:27:58.124474    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d5fef26-c31b-42b0-9015-06c78b08c5b3" path="/var/lib/kubelet/pods/7d5fef26-c31b-42b0-9015-06c78b08c5b3/volumes"
Jul 27 19:28:19.750049 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:19.750011    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9"]
Jul 27 19:28:19.750467 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:19.750261    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9" podUID="6987ede9-ad85-48df-b047-5eaf2e65b1cc" containerName="maas-api" containerID="cri-o://3ec2d370f5cd742d09b70e567efbaeb604359a87a8566fb5c3c127c01f1d9163" gracePeriod=30
Jul 27 19:28:20.004770 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:20.004713    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9"
Jul 27 19:28:20.036394 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:20.036358    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srwtg\" (UniqueName: \"kubernetes.io/projected/6987ede9-ad85-48df-b047-5eaf2e65b1cc-kube-api-access-srwtg\") pod \"6987ede9-ad85-48df-b047-5eaf2e65b1cc\" (UID: \"6987ede9-ad85-48df-b047-5eaf2e65b1cc\") "
Jul 27 19:28:20.036567 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:20.036422    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6987ede9-ad85-48df-b047-5eaf2e65b1cc-openshift-service-ca\") pod \"6987ede9-ad85-48df-b047-5eaf2e65b1cc\" (UID: \"6987ede9-ad85-48df-b047-5eaf2e65b1cc\") "
Jul 27 19:28:20.036567 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:20.036498    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6987ede9-ad85-48df-b047-5eaf2e65b1cc-maas-api-tls\") pod \"6987ede9-ad85-48df-b047-5eaf2e65b1cc\" (UID: \"6987ede9-ad85-48df-b047-5eaf2e65b1cc\") "
Jul 27 19:28:20.036885 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:20.036851    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6987ede9-ad85-48df-b047-5eaf2e65b1cc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6987ede9-ad85-48df-b047-5eaf2e65b1cc" (UID: "6987ede9-ad85-48df-b047-5eaf2e65b1cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:28:20.038863 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:20.038824    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6987ede9-ad85-48df-b047-5eaf2e65b1cc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6987ede9-ad85-48df-b047-5eaf2e65b1cc" (UID: "6987ede9-ad85-48df-b047-5eaf2e65b1cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:28:20.039083 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:20.039057    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6987ede9-ad85-48df-b047-5eaf2e65b1cc-kube-api-access-srwtg" (OuterVolumeSpecName: "kube-api-access-srwtg") pod "6987ede9-ad85-48df-b047-5eaf2e65b1cc" (UID: "6987ede9-ad85-48df-b047-5eaf2e65b1cc"). InnerVolumeSpecName "kube-api-access-srwtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:28:20.137174 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:20.137150    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6987ede9-ad85-48df-b047-5eaf2e65b1cc-openshift-service-ca\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:28:20.137174 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:20.137174    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6987ede9-ad85-48df-b047-5eaf2e65b1cc-maas-api-tls\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:28:20.137311 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:20.137183    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srwtg\" (UniqueName: \"kubernetes.io/projected/6987ede9-ad85-48df-b047-5eaf2e65b1cc-kube-api-access-srwtg\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:28:20.304102 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:20.304064    2566 generic.go:356] "Generic (PLEG): container finished" podID="6987ede9-ad85-48df-b047-5eaf2e65b1cc" containerID="3ec2d370f5cd742d09b70e567efbaeb604359a87a8566fb5c3c127c01f1d9163" exitCode=0
Jul 27 19:28:20.304303 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:20.304108    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9"
Jul 27 19:28:20.304303 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:20.304104    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9" event={"ID":"6987ede9-ad85-48df-b047-5eaf2e65b1cc","Type":"ContainerDied","Data":"3ec2d370f5cd742d09b70e567efbaeb604359a87a8566fb5c3c127c01f1d9163"}
Jul 27 19:28:20.304303 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:20.304181    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9" event={"ID":"6987ede9-ad85-48df-b047-5eaf2e65b1cc","Type":"ContainerDied","Data":"5b266f9bcb30527f6b9593f4f8c24e88de9d62023bdcf4600619278bfb0322e6"}
Jul 27 19:28:20.304303 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:20.304203    2566 scope.go:117] "RemoveContainer" containerID="3ec2d370f5cd742d09b70e567efbaeb604359a87a8566fb5c3c127c01f1d9163"
Jul 27 19:28:20.313584 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:20.313569    2566 scope.go:117] "RemoveContainer" containerID="3ec2d370f5cd742d09b70e567efbaeb604359a87a8566fb5c3c127c01f1d9163"
Jul 27 19:28:20.313845 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:28:20.313824    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ec2d370f5cd742d09b70e567efbaeb604359a87a8566fb5c3c127c01f1d9163\": container with ID starting with 3ec2d370f5cd742d09b70e567efbaeb604359a87a8566fb5c3c127c01f1d9163 not found: ID does not exist" containerID="3ec2d370f5cd742d09b70e567efbaeb604359a87a8566fb5c3c127c01f1d9163"
Jul 27 19:28:20.313894 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:20.313853    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ec2d370f5cd742d09b70e567efbaeb604359a87a8566fb5c3c127c01f1d9163"} err="failed to get container status \"3ec2d370f5cd742d09b70e567efbaeb604359a87a8566fb5c3c127c01f1d9163\": rpc error: code = NotFound desc = could not find container \"3ec2d370f5cd742d09b70e567efbaeb604359a87a8566fb5c3c127c01f1d9163\": container with ID starting with 3ec2d370f5cd742d09b70e567efbaeb604359a87a8566fb5c3c127c01f1d9163 not found: ID does not exist"
Jul 27 19:28:20.335051 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:20.335022    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9"]
Jul 27 19:28:20.340255 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:20.340232    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7f7473-567cf9d4b6-mk7t9"]
Jul 27 19:28:22.123774 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:22.123730    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6987ede9-ad85-48df-b047-5eaf2e65b1cc" path="/var/lib/kubelet/pods/6987ede9-ad85-48df-b047-5eaf2e65b1cc/volumes"
Jul 27 19:28:47.827577 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:47.827542    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt"]
Jul 27 19:28:47.828018 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:47.827786    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt" podUID="14a2e734-99a5-4d2f-9f23-8f8676ba2111" containerName="maas-api" containerID="cri-o://e264cf049bff9ffbc7a521abe826749b0f543ab70faf5a7ecb4c15af8e35d2c1" gracePeriod=30
Jul 27 19:28:48.079715 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:48.079661    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt"
Jul 27 19:28:48.195700 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:48.195670    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2z4hb\" (UniqueName: \"kubernetes.io/projected/14a2e734-99a5-4d2f-9f23-8f8676ba2111-kube-api-access-2z4hb\") pod \"14a2e734-99a5-4d2f-9f23-8f8676ba2111\" (UID: \"14a2e734-99a5-4d2f-9f23-8f8676ba2111\") "
Jul 27 19:28:48.195700 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:48.195699    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14a2e734-99a5-4d2f-9f23-8f8676ba2111-maas-api-tls\") pod \"14a2e734-99a5-4d2f-9f23-8f8676ba2111\" (UID: \"14a2e734-99a5-4d2f-9f23-8f8676ba2111\") "
Jul 27 19:28:48.195891 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:48.195769    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14a2e734-99a5-4d2f-9f23-8f8676ba2111-openshift-service-ca\") pod \"14a2e734-99a5-4d2f-9f23-8f8676ba2111\" (UID: \"14a2e734-99a5-4d2f-9f23-8f8676ba2111\") "
Jul 27 19:28:48.196198 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:48.196167    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a2e734-99a5-4d2f-9f23-8f8676ba2111-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "14a2e734-99a5-4d2f-9f23-8f8676ba2111" (UID: "14a2e734-99a5-4d2f-9f23-8f8676ba2111"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:28:48.197946 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:48.197925    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a2e734-99a5-4d2f-9f23-8f8676ba2111-kube-api-access-2z4hb" (OuterVolumeSpecName: "kube-api-access-2z4hb") pod "14a2e734-99a5-4d2f-9f23-8f8676ba2111" (UID: "14a2e734-99a5-4d2f-9f23-8f8676ba2111"). InnerVolumeSpecName "kube-api-access-2z4hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:28:48.197946 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:48.197929    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14a2e734-99a5-4d2f-9f23-8f8676ba2111-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "14a2e734-99a5-4d2f-9f23-8f8676ba2111" (UID: "14a2e734-99a5-4d2f-9f23-8f8676ba2111"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:28:48.296930 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:48.296904    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z4hb\" (UniqueName: \"kubernetes.io/projected/14a2e734-99a5-4d2f-9f23-8f8676ba2111-kube-api-access-2z4hb\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:28:48.296930 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:48.296929    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14a2e734-99a5-4d2f-9f23-8f8676ba2111-maas-api-tls\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:28:48.297132 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:48.296941    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14a2e734-99a5-4d2f-9f23-8f8676ba2111-openshift-service-ca\") on node \"ip-10-0-128-196.ec2.internal\" DevicePath \"\""
Jul 27 19:28:48.460043 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:48.459957    2566 generic.go:356] "Generic (PLEG): container finished" podID="14a2e734-99a5-4d2f-9f23-8f8676ba2111" containerID="e264cf049bff9ffbc7a521abe826749b0f543ab70faf5a7ecb4c15af8e35d2c1" exitCode=0
Jul 27 19:28:48.460043 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:48.460002    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt"
Jul 27 19:28:48.460043 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:48.460031    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt" event={"ID":"14a2e734-99a5-4d2f-9f23-8f8676ba2111","Type":"ContainerDied","Data":"e264cf049bff9ffbc7a521abe826749b0f543ab70faf5a7ecb4c15af8e35d2c1"}
Jul 27 19:28:48.460312 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:48.460071    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt" event={"ID":"14a2e734-99a5-4d2f-9f23-8f8676ba2111","Type":"ContainerDied","Data":"3d0ab9e821606be588c0fc87aa64d237b9e45fa7942b7e4105f873b22a6653d4"}
Jul 27 19:28:48.460312 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:48.460094    2566 scope.go:117] "RemoveContainer" containerID="e264cf049bff9ffbc7a521abe826749b0f543ab70faf5a7ecb4c15af8e35d2c1"
Jul 27 19:28:48.470136 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:48.470103    2566 scope.go:117] "RemoveContainer" containerID="e264cf049bff9ffbc7a521abe826749b0f543ab70faf5a7ecb4c15af8e35d2c1"
Jul 27 19:28:48.470366 ip-10-0-128-196 kubenswrapper[2566]: E0727 19:28:48.470345    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e264cf049bff9ffbc7a521abe826749b0f543ab70faf5a7ecb4c15af8e35d2c1\": container with ID starting with e264cf049bff9ffbc7a521abe826749b0f543ab70faf5a7ecb4c15af8e35d2c1 not found: ID does not exist" containerID="e264cf049bff9ffbc7a521abe826749b0f543ab70faf5a7ecb4c15af8e35d2c1"
Jul 27 19:28:48.470460 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:48.470371    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e264cf049bff9ffbc7a521abe826749b0f543ab70faf5a7ecb4c15af8e35d2c1"} err="failed to get container status \"e264cf049bff9ffbc7a521abe826749b0f543ab70faf5a7ecb4c15af8e35d2c1\": rpc error: code = NotFound desc = could not find container \"e264cf049bff9ffbc7a521abe826749b0f543ab70faf5a7ecb4c15af8e35d2c1\": container with ID starting with e264cf049bff9ffbc7a521abe826749b0f543ab70faf5a7ecb4c15af8e35d2c1 not found: ID does not exist"
Jul 27 19:28:48.487286 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:48.487264    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt"]
Jul 27 19:28:48.500913 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:48.500891    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6710a9-79c46655cf-cdtkt"]
Jul 27 19:28:50.124363 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:50.124288    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14a2e734-99a5-4d2f-9f23-8f8676ba2111" path="/var/lib/kubelet/pods/14a2e734-99a5-4d2f-9f23-8f8676ba2111/volumes"
Jul 27 19:28:59.370431 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:28:59.370384    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7dc5bb56d-js2x6_facc30c3-4d3f-4939-8568-4581518e23a2/authorino/0.log"
Jul 27 19:29:00.400867 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:00.400841    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 19:29:00.417949 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:00.417921    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdb9t_fa213400-5979-496e-9d82-f6ba6a0f0333/ovn-acl-logging/0.log"
Jul 27 19:29:06.479241 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:06.479209    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57d99c84d4-6x7ln_f980df40-310c-4e8b-a9d7-f52ab379e938/copy-manifests/0.log"
Jul 27 19:29:06.497692 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:06.497660    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57d99c84d4-6x7ln_f980df40-310c-4e8b-a9d7-f52ab379e938/manager/0.log"
Jul 27 19:29:07.015666 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:07.015632    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-557776f786-xvfwc_af0c2f08-b3af-4f51-944b-22262f3f7e18/maas-api/0.log"
Jul 27 19:29:07.972868 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:07.972778    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx_9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a/util/0.log"
Jul 27 19:29:07.985503 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:07.985475    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx_9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a/pull/0.log"
Jul 27 19:29:07.993592 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:07.993571    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx_9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a/extract/0.log"
Jul 27 19:29:08.112351 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:08.112323    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7_a0ccd514-6a2b-405f-a724-297b04ae55a7/pull/0.log"
Jul 27 19:29:08.121841 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:08.121816    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7_a0ccd514-6a2b-405f-a724-297b04ae55a7/extract/0.log"
Jul 27 19:29:08.139504 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:08.139487    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7_a0ccd514-6a2b-405f-a724-297b04ae55a7/util/0.log"
Jul 27 19:29:08.276142 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:08.276094    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv_b058ecf5-cd45-4a3d-bbc2-733a137142ec/util/0.log"
Jul 27 19:29:08.291684 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:08.291657    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv_b058ecf5-cd45-4a3d-bbc2-733a137142ec/pull/0.log"
Jul 27 19:29:08.301510 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:08.301481    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv_b058ecf5-cd45-4a3d-bbc2-733a137142ec/extract/0.log"
Jul 27 19:29:08.423623 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:08.423604    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r_d3586023-5e0a-4e8a-b384-0d17e7337513/util/0.log"
Jul 27 19:29:08.434303 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:08.434277    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r_d3586023-5e0a-4e8a-b384-0d17e7337513/pull/0.log"
Jul 27 19:29:08.443949 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:08.443927    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r_d3586023-5e0a-4e8a-b384-0d17e7337513/extract/0.log"
Jul 27 19:29:08.577176 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:08.577101    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7dc5bb56d-js2x6_facc30c3-4d3f-4939-8568-4581518e23a2/authorino/0.log"
Jul 27 19:29:09.518357 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:09.518328    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-m84gm_972972f9-1bef-4c24-8b1a-0c6076d73dab/manager/0.log"
Jul 27 19:29:10.118619 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:10.118587    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7896769b64-dmt58_445cd3bc-555e-47cf-99c7-ca02cdd89d85/kube-auth-proxy/0.log"
Jul 27 19:29:10.507711 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:10.507625    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-4jtjz_ebee721b-f5ea-448d-adf8-5cf2398fa566/payload-pre-processing/0.log"
Jul 27 19:29:11.423260 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:11.423229    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-46ggf_aef97511-8f66-4ab2-9b17-a0b0b10b0ff6/main/0.log"
Jul 27 19:29:11.813955 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:11.813927    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-gvnzw_27781d28-5639-4a2e-9cd9-e3f5804fa0ca/main/0.log"
Jul 27 19:29:18.097232 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.097198    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pshc7/must-gather-4rlwt"]
Jul 27 19:29:18.097608 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.097585    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6635a8a-0116-40b1-b108-17e1aaad2f44" containerName="main"
Jul 27 19:29:18.097608 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.097598    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6635a8a-0116-40b1-b108-17e1aaad2f44" containerName="main"
Jul 27 19:29:18.097686 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.097612    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d5fef26-c31b-42b0-9015-06c78b08c5b3" containerName="maas-api"
Jul 27 19:29:18.097686 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.097618    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d5fef26-c31b-42b0-9015-06c78b08c5b3" containerName="maas-api"
Jul 27 19:29:18.097686 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.097625    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b1616d3-ff3a-48a6-b783-e486810a0545" containerName="maas-api"
Jul 27 19:29:18.097686 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.097631    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b1616d3-ff3a-48a6-b783-e486810a0545" containerName="maas-api"
Jul 27 19:29:18.097686 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.097641    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6987ede9-ad85-48df-b047-5eaf2e65b1cc" containerName="maas-api"
Jul 27 19:29:18.097686 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.097646    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6987ede9-ad85-48df-b047-5eaf2e65b1cc" containerName="maas-api"
Jul 27 19:29:18.097686 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.097661    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14a2e734-99a5-4d2f-9f23-8f8676ba2111" containerName="maas-api"
Jul 27 19:29:18.097686 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.097666    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="14a2e734-99a5-4d2f-9f23-8f8676ba2111" containerName="maas-api"
Jul 27 19:29:18.097686 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.097676    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e13dde1c-97a3-4284-ab61-e15c321893d1" containerName="maas-api"
Jul 27 19:29:18.097686 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.097681    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e13dde1c-97a3-4284-ab61-e15c321893d1" containerName="maas-api"
Jul 27 19:29:18.097984 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.097740    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6987ede9-ad85-48df-b047-5eaf2e65b1cc" containerName="maas-api"
Jul 27 19:29:18.097984 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.097748    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="14a2e734-99a5-4d2f-9f23-8f8676ba2111" containerName="maas-api"
Jul 27 19:29:18.097984 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.097755    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6635a8a-0116-40b1-b108-17e1aaad2f44" containerName="main"
Jul 27 19:29:18.097984 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.097762    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b1616d3-ff3a-48a6-b783-e486810a0545" containerName="maas-api"
Jul 27 19:29:18.097984 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.097771    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d5fef26-c31b-42b0-9015-06c78b08c5b3" containerName="maas-api"
Jul 27 19:29:18.097984 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.097776    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e13dde1c-97a3-4284-ab61-e15c321893d1" containerName="maas-api"
Jul 27 19:29:18.100857 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.100841    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pshc7/must-gather-4rlwt"
Jul 27 19:29:18.110670 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.110649    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pshc7\"/\"kube-root-ca.crt\""
Jul 27 19:29:18.118563 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.118543    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pshc7\"/\"openshift-service-ca.crt\""
Jul 27 19:29:18.128968 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.128945    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pshc7/must-gather-4rlwt"]
Jul 27 19:29:18.171994 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.171961    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/45fd58a5-e7c1-43a3-b3d5-d46bb90d8ccf-must-gather-output\") pod \"must-gather-4rlwt\" (UID: \"45fd58a5-e7c1-43a3-b3d5-d46bb90d8ccf\") " pod="openshift-must-gather-pshc7/must-gather-4rlwt"
Jul 27 19:29:18.172177 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.172048    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftbnz\" (UniqueName: \"kubernetes.io/projected/45fd58a5-e7c1-43a3-b3d5-d46bb90d8ccf-kube-api-access-ftbnz\") pod \"must-gather-4rlwt\" (UID: \"45fd58a5-e7c1-43a3-b3d5-d46bb90d8ccf\") " pod="openshift-must-gather-pshc7/must-gather-4rlwt"
Jul 27 19:29:18.272872 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.272833    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/45fd58a5-e7c1-43a3-b3d5-d46bb90d8ccf-must-gather-output\") pod \"must-gather-4rlwt\" (UID: \"45fd58a5-e7c1-43a3-b3d5-d46bb90d8ccf\") " pod="openshift-must-gather-pshc7/must-gather-4rlwt"
Jul 27 19:29:18.273075 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.272935    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftbnz\" (UniqueName: \"kubernetes.io/projected/45fd58a5-e7c1-43a3-b3d5-d46bb90d8ccf-kube-api-access-ftbnz\") pod \"must-gather-4rlwt\" (UID: \"45fd58a5-e7c1-43a3-b3d5-d46bb90d8ccf\") " pod="openshift-must-gather-pshc7/must-gather-4rlwt"
Jul 27 19:29:18.273228 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.273206    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/45fd58a5-e7c1-43a3-b3d5-d46bb90d8ccf-must-gather-output\") pod \"must-gather-4rlwt\" (UID: \"45fd58a5-e7c1-43a3-b3d5-d46bb90d8ccf\") " pod="openshift-must-gather-pshc7/must-gather-4rlwt"
Jul 27 19:29:18.305616 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.305581    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftbnz\" (UniqueName: \"kubernetes.io/projected/45fd58a5-e7c1-43a3-b3d5-d46bb90d8ccf-kube-api-access-ftbnz\") pod \"must-gather-4rlwt\" (UID: \"45fd58a5-e7c1-43a3-b3d5-d46bb90d8ccf\") " pod="openshift-must-gather-pshc7/must-gather-4rlwt"
Jul 27 19:29:18.410223 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.410097    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pshc7/must-gather-4rlwt"
Jul 27 19:29:18.577918 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.577872    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pshc7/must-gather-4rlwt"]
Jul 27 19:29:18.583971 ip-10-0-128-196 kubenswrapper[2566]: W0727 19:29:18.583946    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45fd58a5_e7c1_43a3_b3d5_d46bb90d8ccf.slice/crio-8fc2622476fa60a9311ef6d75a3ff957824d958bf89e7b609331c9dbcd5ce5fe WatchSource:0}: Error finding container 8fc2622476fa60a9311ef6d75a3ff957824d958bf89e7b609331c9dbcd5ce5fe: Status 404 returned error can't find the container with id 8fc2622476fa60a9311ef6d75a3ff957824d958bf89e7b609331c9dbcd5ce5fe
Jul 27 19:29:18.585612 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.585595    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 19:29:18.625136 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:18.625093    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pshc7/must-gather-4rlwt" event={"ID":"45fd58a5-e7c1-43a3-b3d5-d46bb90d8ccf","Type":"ContainerStarted","Data":"8fc2622476fa60a9311ef6d75a3ff957824d958bf89e7b609331c9dbcd5ce5fe"}
Jul 27 19:29:19.634979 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:19.634910    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pshc7/must-gather-4rlwt" event={"ID":"45fd58a5-e7c1-43a3-b3d5-d46bb90d8ccf","Type":"ContainerStarted","Data":"dc4e72f35e913cfad2b739f3490c50ed4aa49fa35fca09646871df9eb313105b"}
Jul 27 19:29:20.643165 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:20.643097    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pshc7/must-gather-4rlwt" event={"ID":"45fd58a5-e7c1-43a3-b3d5-d46bb90d8ccf","Type":"ContainerStarted","Data":"7f04064cccb93b07d216fdeb2ea5bc290af93cde894634d8e58ce0efcb1d97e2"}
Jul 27 19:29:20.670505 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:20.670436    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pshc7/must-gather-4rlwt" podStartSLOduration=1.767793489 podStartE2EDuration="2.670413062s" totalImagesPullingTime="902.619573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:29:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:29:20.664406985 +0000 UTC m=+3621.098664596" watchObservedRunningTime="2026-07-27 19:29:20.670413062 +0000 UTC m=+3621.104670677"
Jul 27 19:29:21.332302 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:21.332272    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qrfwk_662aa4f5-23c0-42d2-8b21-270c4b33e9a5/global-pull-secret-syncer/0.log"
Jul 27 19:29:21.496627 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:21.496588    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l6rrw_5b7a8e0f-9255-415f-b6fe-123c07105bc3/konnectivity-agent/0.log"
Jul 27 19:29:21.567162 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:21.567107    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-196.ec2.internal_7cd8d2439d53474a3dfd84498491b9ab/haproxy/0.log"
Jul 27 19:29:25.213310 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:25.213277    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx_9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a/extract/0.log"
Jul 27 19:29:25.259299 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:25.259264    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx_9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a/util/0.log"
Jul 27 19:29:25.295491 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:25.295457    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sfdjx_9d6de99c-69ea-4fcc-8fc2-8a2d23c3cf6a/pull/0.log"
Jul 27 19:29:25.350030 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:25.349995    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7_a0ccd514-6a2b-405f-a724-297b04ae55a7/extract/0.log"
Jul 27 19:29:25.384798 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:25.384767    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7_a0ccd514-6a2b-405f-a724-297b04ae55a7/util/0.log"
Jul 27 19:29:25.420874 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:25.420819    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hblv7_a0ccd514-6a2b-405f-a724-297b04ae55a7/pull/0.log"
Jul 27 19:29:25.467581 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:25.467498    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv_b058ecf5-cd45-4a3d-bbc2-733a137142ec/extract/0.log"
Jul 27 19:29:25.505521 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:25.505489    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv_b058ecf5-cd45-4a3d-bbc2-733a137142ec/util/0.log"
Jul 27 19:29:25.551321 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:25.551286    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qt7nv_b058ecf5-cd45-4a3d-bbc2-733a137142ec/pull/0.log"
Jul 27 19:29:25.594655 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:25.594622    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r_d3586023-5e0a-4e8a-b384-0d17e7337513/extract/0.log"
Jul 27 19:29:25.632719 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:25.632687    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r_d3586023-5e0a-4e8a-b384-0d17e7337513/util/0.log"
Jul 27 19:29:25.677919 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:25.677884    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gpg9r_d3586023-5e0a-4e8a-b384-0d17e7337513/pull/0.log"
Jul 27 19:29:26.341195 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:26.341156    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7dc5bb56d-js2x6_facc30c3-4d3f-4939-8568-4581518e23a2/authorino/0.log"
Jul 27 19:29:27.301461 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:27.301426    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-m84gm_972972f9-1bef-4c24-8b1a-0c6076d73dab/manager/0.log"
Jul 27 19:29:29.133351 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:29.133322    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd1fb1e4-c04c-45bc-91ce-4df135cb07f0/alertmanager/0.log"
Jul 27 19:29:29.239187 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:29.239150    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd1fb1e4-c04c-45bc-91ce-4df135cb07f0/config-reloader/0.log"
Jul 27 19:29:29.277896 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:29.277866    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd1fb1e4-c04c-45bc-91ce-4df135cb07f0/kube-rbac-proxy-web/0.log"
Jul 27 19:29:29.375258 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:29.375226    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd1fb1e4-c04c-45bc-91ce-4df135cb07f0/kube-rbac-proxy/0.log"
Jul 27 19:29:29.417194 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:29.417090    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd1fb1e4-c04c-45bc-91ce-4df135cb07f0/kube-rbac-proxy-metric/0.log"
Jul 27 19:29:29.456059 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:29.456035    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd1fb1e4-c04c-45bc-91ce-4df135cb07f0/prom-label-proxy/0.log"
Jul 27 19:29:29.494432 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:29.494394    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd1fb1e4-c04c-45bc-91ce-4df135cb07f0/init-config-reloader/0.log"
Jul 27 19:29:29.601012 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:29.600982    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-jpfpd_df58afda-7805-4559-a9ad-e5526362597b/kube-state-metrics/0.log"
Jul 27 19:29:29.642863 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:29.642829    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-jpfpd_df58afda-7805-4559-a9ad-e5526362597b/kube-rbac-proxy-main/0.log"
Jul 27 19:29:29.677612 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:29.677527    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-jpfpd_df58afda-7805-4559-a9ad-e5526362597b/kube-rbac-proxy-self/0.log"
Jul 27 19:29:29.829488 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:29.829457    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jb69h_b5e71958-e291-4c2d-ba62-31080dbe1a2d/node-exporter/0.log"
Jul 27 19:29:29.865367 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:29.865335    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jb69h_b5e71958-e291-4c2d-ba62-31080dbe1a2d/kube-rbac-proxy/0.log"
Jul 27 19:29:29.903576 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:29.903549    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jb69h_b5e71958-e291-4c2d-ba62-31080dbe1a2d/init-textfile/0.log"
Jul 27 19:29:30.159111 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:30.159080    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-2cgng_6ffe2667-dc91-42e8-b89a-b8a06e28ecc0/kube-rbac-proxy-main/0.log"
Jul 27 19:29:30.186950 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:30.186911    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-2cgng_6ffe2667-dc91-42e8-b89a-b8a06e28ecc0/kube-rbac-proxy-self/0.log"
Jul 27 19:29:30.217358 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:30.217320    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-2cgng_6ffe2667-dc91-42e8-b89a-b8a06e28ecc0/openshift-state-metrics/0.log"
Jul 27 19:29:30.625706 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:30.625667    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-shhvl_2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50/prometheus-operator/0.log"
Jul 27 19:29:30.701439 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:30.701411    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-shhvl_2d7aefd4-d2b5-4a69-beb7-f2d0dbe7da50/kube-rbac-proxy/0.log"
Jul 27 19:29:30.765107 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:30.765078    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-hfg4n_8c0ca6e3-528f-4cd5-841c-4ab0ccedf99f/prometheus-operator-admission-webhook/0.log"
Jul 27 19:29:30.824966 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:30.824935    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5588ddd8b5-6dg78_6ebc1e23-4641-4e18-ac64-a2b7c605492e/telemeter-client/0.log"
Jul 27 19:29:30.870227 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:30.870196    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5588ddd8b5-6dg78_6ebc1e23-4641-4e18-ac64-a2b7c605492e/reload/0.log"
Jul 27 19:29:30.928647 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:30.918439    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5588ddd8b5-6dg78_6ebc1e23-4641-4e18-ac64-a2b7c605492e/kube-rbac-proxy/0.log"
Jul 27 19:29:30.962979 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:30.962947    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-698c5745b9-vmg67_67d33c62-ae92-4941-b009-b77e3e93a2c2/thanos-query/0.log"
Jul 27 19:29:30.998499 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:30.998466    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-698c5745b9-vmg67_67d33c62-ae92-4941-b009-b77e3e93a2c2/kube-rbac-proxy-web/0.log"
Jul 27 19:29:31.037029 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:31.036998    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-698c5745b9-vmg67_67d33c62-ae92-4941-b009-b77e3e93a2c2/kube-rbac-proxy/0.log"
Jul 27 19:29:31.074656 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:31.074620    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-698c5745b9-vmg67_67d33c62-ae92-4941-b009-b77e3e93a2c2/prom-label-proxy/0.log"
Jul 27 19:29:31.128657 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:31.128626    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-698c5745b9-vmg67_67d33c62-ae92-4941-b009-b77e3e93a2c2/kube-rbac-proxy-rules/0.log"
Jul 27 19:29:31.162497 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:31.162466    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pshc7/perf-node-gather-daemonset-7bs66"]
Jul 27 19:29:31.166759 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:31.166734    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-7bs66"
Jul 27 19:29:31.171678 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:31.171654    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pshc7\"/\"default-dockercfg-55v4q\""
Jul 27 19:29:31.187263 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:31.187190    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pshc7/perf-node-gather-daemonset-7bs66"]
Jul 27 19:29:31.196103 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:31.196074    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-698c5745b9-vmg67_67d33c62-ae92-4941-b009-b77e3e93a2c2/kube-rbac-proxy-metrics/0.log"
Jul 27 19:29:31.212062 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:31.212029    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnzx4\" (UniqueName: \"kubernetes.io/projected/97a27ee6-3678-4046-b28e-47ca38e96095-kube-api-access-tnzx4\") pod \"perf-node-gather-daemonset-7bs66\" (UID: \"97a27ee6-3678-4046-b28e-47ca38e96095\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-7bs66"
Jul 27 19:29:31.212281 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:31.212097    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/97a27ee6-3678-4046-b28e-47ca38e96095-proc\") pod \"perf-node-gather-daemonset-7bs66\" (UID: \"97a27ee6-3678-4046-b28e-47ca38e96095\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-7bs66"
Jul 27 19:29:31.212281 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:31.212230    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/97a27ee6-3678-4046-b28e-47ca38e96095-podres\") pod \"perf-node-gather-daemonset-7bs66\" (UID: \"97a27ee6-3678-4046-b28e-47ca38e96095\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-7bs66"
Jul 27 19:29:31.212396 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:31.212284    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97a27ee6-3678-4046-b28e-47ca38e96095-lib-modules\") pod \"perf-node-gather-daemonset-7bs66\" (UID: \"97a27ee6-3678-4046-b28e-47ca38e96095\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-7bs66"
Jul 27 19:29:31.212396 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:31.212378    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97a27ee6-3678-4046-b28e-47ca38e96095-sys\") pod \"perf-node-gather-daemonset-7bs66\" (UID: \"97a27ee6-3678-4046-b28e-47ca38e96095\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-7bs66"
Jul 27 19:29:31.313001 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:31.312930    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/97a27ee6-3678-4046-b28e-47ca38e96095-proc\") pod \"perf-node-gather-daemonset-7bs66\" (UID: \"97a27ee6-3678-4046-b28e-47ca38e96095\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-7bs66"
Jul 27 19:29:31.313227 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:31.313032    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/97a27ee6-3678-4046-b28e-47ca38e96095-podres\") pod \"perf-node-gather-daemonset-7bs66\" (UID: \"97a27ee6-3678-4046-b28e-47ca38e96095\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-7bs66"
Jul 27 19:29:31.313227 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:31.313082    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97a27ee6-3678-4046-b28e-47ca38e96095-lib-modules\") pod \"perf-node-gather-daemonset-7bs66\" (UID: \"97a27ee6-3678-4046-b28e-47ca38e96095\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-7bs66"
Jul 27 19:29:31.313227 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:31.313182    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97a27ee6-3678-4046-b28e-47ca38e96095-sys\") pod \"perf-node-gather-daemonset-7bs66\" (UID: \"97a27ee6-3678-4046-b28e-47ca38e96095\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-7bs66"
Jul 27 19:29:31.313227 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:31.313224    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnzx4\" (UniqueName: \"kubernetes.io/projected/97a27ee6-3678-4046-b28e-47ca38e96095-kube-api-access-tnzx4\") pod \"perf-node-gather-daemonset-7bs66\" (UID: \"97a27ee6-3678-4046-b28e-47ca38e96095\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-7bs66"
Jul 27 19:29:31.313465 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:31.313345    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/97a27ee6-3678-4046-b28e-47ca38e96095-podres\") pod \"perf-node-gather-daemonset-7bs66\" (UID: \"97a27ee6-3678-4046-b28e-47ca38e96095\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-7bs66"
Jul 27 19:29:31.313465 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:31.313399    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/97a27ee6-3678-4046-b28e-47ca38e96095-proc\") pod \"perf-node-gather-daemonset-7bs66\" (UID: \"97a27ee6-3678-4046-b28e-47ca38e96095\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-7bs66"
Jul 27 19:29:31.313465 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:31.313454    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97a27ee6-3678-4046-b28e-47ca38e96095-sys\") pod \"perf-node-gather-daemonset-7bs66\" (UID: \"97a27ee6-3678-4046-b28e-47ca38e96095\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-7bs66"
Jul 27 19:29:31.313599 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:31.313460    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97a27ee6-3678-4046-b28e-47ca38e96095-lib-modules\") pod \"perf-node-gather-daemonset-7bs66\" (UID: \"97a27ee6-3678-4046-b28e-47ca38e96095\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-7bs66"
Jul 27 19:29:31.326860 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:31.324342    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnzx4\" (UniqueName: \"kubernetes.io/projected/97a27ee6-3678-4046-b28e-47ca38e96095-kube-api-access-tnzx4\") pod \"perf-node-gather-daemonset-7bs66\" (UID: \"97a27ee6-3678-4046-b28e-47ca38e96095\") " pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-7bs66"
Jul 27 19:29:31.480368 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:31.480280    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-7bs66"
Jul 27 19:29:31.660104 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:31.657209    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pshc7/perf-node-gather-daemonset-7bs66"]
Jul 27 19:29:31.666542 ip-10-0-128-196 kubenswrapper[2566]: W0727 19:29:31.666500    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod97a27ee6_3678_4046_b28e_47ca38e96095.slice/crio-12971865bf0ba41fc0d60c3fceb2994a6b916dcb5746f3d72420f022a1c2fcf0 WatchSource:0}: Error finding container 12971865bf0ba41fc0d60c3fceb2994a6b916dcb5746f3d72420f022a1c2fcf0: Status 404 returned error can't find the container with id 12971865bf0ba41fc0d60c3fceb2994a6b916dcb5746f3d72420f022a1c2fcf0
Jul 27 19:29:31.736185 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:31.736097    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-7bs66" event={"ID":"97a27ee6-3678-4046-b28e-47ca38e96095","Type":"ContainerStarted","Data":"12971865bf0ba41fc0d60c3fceb2994a6b916dcb5746f3d72420f022a1c2fcf0"}
Jul 27 19:29:32.746380 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:32.746346    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-7bs66" event={"ID":"97a27ee6-3678-4046-b28e-47ca38e96095","Type":"ContainerStarted","Data":"738ec91a8819c07fa3a9703585ba51fff6968b4b1b192f09321211f2ea195334"}
Jul 27 19:29:32.746830 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:32.746403    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-7bs66"
Jul 27 19:29:32.779156 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:32.779068    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-7bs66" podStartSLOduration=1.779047394 podStartE2EDuration="1.779047394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:29:32.777588316 +0000 UTC m=+3633.211845929" watchObservedRunningTime="2026-07-27 19:29:32.779047394 +0000 UTC m=+3633.213305006"
Jul 27 19:29:33.553348 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:33.553318    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7cb99d6978-pctpl_1103b1ea-cab3-4559-aa88-fd5743f33a8d/console/0.log"
Jul 27 19:29:33.603573 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:33.603543    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-b2b27_0cc5553b-6803-4a64-a3b4-a4ef998a14dc/download-server/0.log"
Jul 27 19:29:35.446442 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:35.446412    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r7j9k_423413aa-a12b-40ce-9da1-02de296eb175/dns/0.log"
Jul 27 19:29:35.473776 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:35.473750    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r7j9k_423413aa-a12b-40ce-9da1-02de296eb175/kube-rbac-proxy/0.log"
Jul 27 19:29:35.506056 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:35.506025    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cl7cw_4378195f-ed99-42aa-90e6-0fde9f5394f1/dns-node-resolver/0.log"
Jul 27 19:29:36.264300 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:36.264262    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4g7qz_0806ed23-3336-4327-ae3a-0da1e826f856/node-ca/0.log"
Jul 27 19:29:37.623791 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:37.623760    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7896769b64-dmt58_445cd3bc-555e-47cf-99c7-ca02cdd89d85/kube-auth-proxy/0.log"
Jul 27 19:29:37.811760 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:37.811726    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-4jtjz_ebee721b-f5ea-448d-adf8-5cf2398fa566/payload-pre-processing/0.log"
Jul 27 19:29:38.766071 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:38.766029    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pshc7/perf-node-gather-daemonset-7bs66"
Jul 27 19:29:39.039248 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:39.039213    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sjj6g_db0becf5-bc5c-4511-8ccd-43c17b98293e/serve-healthcheck-canary/0.log"
Jul 27 19:29:39.862319 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:39.862288    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ft289_d94bb424-b152-4636-9e19-ba2a0a55ceb4/kube-rbac-proxy/0.log"
Jul 27 19:29:39.897892 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:39.897866    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ft289_d94bb424-b152-4636-9e19-ba2a0a55ceb4/exporter/0.log"
Jul 27 19:29:39.947053 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:39.947021    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ft289_d94bb424-b152-4636-9e19-ba2a0a55ceb4/extractor/0.log"
Jul 27 19:29:42.117520 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:42.117479    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7896769b64-dmt58_445cd3bc-555e-47cf-99c7-ca02cdd89d85/kube-auth-proxy/0.log"
Jul 27 19:29:42.252559 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:42.252521    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-4jtjz_ebee721b-f5ea-448d-adf8-5cf2398fa566/payload-pre-processing/0.log"
Jul 27 19:29:43.951562 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:43.951476    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7896769b64-dmt58_445cd3bc-555e-47cf-99c7-ca02cdd89d85/kube-auth-proxy/0.log"
Jul 27 19:29:44.076181 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:44.076105    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57d99c84d4-6x7ln_f980df40-310c-4e8b-a9d7-f52ab379e938/manager/0.log"
Jul 27 19:29:44.099765 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:44.099734    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-4jtjz_ebee721b-f5ea-448d-adf8-5cf2398fa566/payload-pre-processing/0.log"
Jul 27 19:29:44.108943 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:44.108917    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57d99c84d4-6x7ln_f980df40-310c-4e8b-a9d7-f52ab379e938/copy-manifests/0.log"
Jul 27 19:29:45.544542 ip-10-0-128-196 kubenswrapper[2566]: I0727 19:29:45.544497    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7cc45b746b-rwfq7_0c986fb5-995a-4218-9879-19baef748503/manager/0.log"