Jul 17 10:57:24.956264 ip-10-0-140-233 systemd[1]: Starting Kubernetes Kubelet...
Jul 17 10:57:25.347637 ip-10-0-140-233 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 10:57:25.347637 ip-10-0-140-233 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 17 10:57:25.347637 ip-10-0-140-233 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 10:57:25.347637 ip-10-0-140-233 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 17 10:57:25.347637 ip-10-0-140-233 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 10:57:25.347637 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.343750    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 17 10:57:25.460010 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348563    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 10:57:25.460010 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348575    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 10:57:25.460010 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348578    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 10:57:25.460010 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348582    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 10:57:25.460010 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348585    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 10:57:25.460010 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348588    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 10:57:25.460010 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348591    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 10:57:25.460010 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348594    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 10:57:25.460010 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348597    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 10:57:25.460010 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348599    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 10:57:25.460010 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348602    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 10:57:25.460010 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348604    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 10:57:25.460010 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348607    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 10:57:25.460010 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348613    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 10:57:25.460010 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348615    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 10:57:25.460010 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348620    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 10:57:25.460010 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348624    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 10:57:25.460010 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348627    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 10:57:25.460010 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348630    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 10:57:25.420651 ip-10-0-140-233 systemd[1]: Started Kubernetes Kubelet.
Jul 17 10:57:25.461087 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348633    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 10:57:25.461087 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348635    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 10:57:25.461087 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348638    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 10:57:25.461087 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348641    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 10:57:25.461087 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348644    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 10:57:25.461087 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348646    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 10:57:25.461087 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348649    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 10:57:25.461087 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348651    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 10:57:25.461087 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348654    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 10:57:25.461087 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348656    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 10:57:25.461087 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348659    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 10:57:25.461087 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348661    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 10:57:25.461087 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348664    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 10:57:25.461087 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348667    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 10:57:25.461087 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348670    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 10:57:25.461087 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348673    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 10:57:25.461087 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348675    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 10:57:25.461087 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348678    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 10:57:25.461087 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348681    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 10:57:25.461087 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348684    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 10:57:25.463021 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348686    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 10:57:25.463021 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348688    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 10:57:25.463021 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348691    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 10:57:25.463021 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348694    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 10:57:25.463021 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348696    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 10:57:25.463021 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348699    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 10:57:25.463021 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348701    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 10:57:25.463021 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348705    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 10:57:25.463021 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348707    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 10:57:25.463021 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348710    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 10:57:25.463021 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348712    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 10:57:25.463021 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348714    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 10:57:25.463021 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348717    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 10:57:25.463021 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348719    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 10:57:25.463021 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348722    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 10:57:25.463021 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348724    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 10:57:25.463021 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348727    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 10:57:25.463021 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348729    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 10:57:25.463021 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348732    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 10:57:25.463021 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348734    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 10:57:25.464979 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348737    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 10:57:25.464979 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348740    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 10:57:25.464979 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348742    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 10:57:25.464979 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348745    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 10:57:25.464979 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348749    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 10:57:25.464979 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348753    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 10:57:25.464979 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348756    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 10:57:25.464979 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348759    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 10:57:25.464979 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348762    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 10:57:25.464979 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348765    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 10:57:25.464979 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348768    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 10:57:25.464979 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348771    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 10:57:25.464979 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348774    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 10:57:25.464979 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348776    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 10:57:25.464979 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348779    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 10:57:25.464979 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348782    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 10:57:25.464979 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348784    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 10:57:25.464979 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348787    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 10:57:25.464979 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348789    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 10:57:25.466416 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348792    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 10:57:25.466416 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348796    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 10:57:25.466416 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348799    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 10:57:25.466416 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348802    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 10:57:25.466416 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348805    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 10:57:25.466416 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348807    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 10:57:25.466416 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348810    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 10:57:25.466416 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348814    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 10:57:25.466416 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348816    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 10:57:25.466416 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348819    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 10:57:25.466416 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348822    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 10:57:25.466416 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348824    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 10:57:25.466416 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348827    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 10:57:25.466416 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348830    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 10:57:25.466416 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348833    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 10:57:25.466416 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348836    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 10:57:25.466416 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348838    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 10:57:25.466416 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348841    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 10:57:25.466416 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348844    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 10:57:25.467404 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348847    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 10:57:25.467404 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.348850    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 10:57:25.467404 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349568    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 10:57:25.467404 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349574    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 10:57:25.467404 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349577    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 10:57:25.467404 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349580    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 10:57:25.467404 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349583    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 10:57:25.467404 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349586    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 10:57:25.467404 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349591    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 10:57:25.467404 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349595    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 10:57:25.467404 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349598    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 10:57:25.467404 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349601    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 10:57:25.467404 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349604    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 10:57:25.467404 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349606    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 10:57:25.467404 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349609    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 10:57:25.467404 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349611    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 10:57:25.467404 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349614    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 10:57:25.467404 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349616    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 10:57:25.467404 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349619    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 10:57:25.468896 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349622    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 10:57:25.468896 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349624    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 10:57:25.468896 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349627    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 10:57:25.468896 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349629    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 10:57:25.468896 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349632    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 10:57:25.468896 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349635    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 10:57:25.468896 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349638    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 10:57:25.468896 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349640    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 10:57:25.468896 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349643    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 10:57:25.468896 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349645    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 10:57:25.468896 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349648    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 10:57:25.468896 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349650    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 10:57:25.468896 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349652    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 10:57:25.468896 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349655    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 10:57:25.468896 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349658    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 10:57:25.468896 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349661    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 10:57:25.468896 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349664    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 10:57:25.468896 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349667    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 10:57:25.468896 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349670    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 10:57:25.471296 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349672    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 10:57:25.471296 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349675    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 10:57:25.471296 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349678    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 10:57:25.471296 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349680    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 10:57:25.471296 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349683    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 10:57:25.471296 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349685    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 10:57:25.471296 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349687    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 10:57:25.471296 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349690    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 10:57:25.471296 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349692    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 10:57:25.471296 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349695    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 10:57:25.471296 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349697    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 10:57:25.471296 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349701    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 10:57:25.471296 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349703    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 10:57:25.471296 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349706    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 10:57:25.471296 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349709    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 10:57:25.471296 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349711    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 10:57:25.471296 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349714    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 10:57:25.471296 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349716    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 10:57:25.471296 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349718    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 10:57:25.471296 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349721    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 10:57:25.471924 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349723    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 10:57:25.471924 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349725    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 10:57:25.471924 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349728    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 10:57:25.471924 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349730    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 10:57:25.471924 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349732    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 10:57:25.471924 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349735    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 10:57:25.471924 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349737    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 10:57:25.471924 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349740    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 10:57:25.471924 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349742    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 10:57:25.471924 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349744    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 10:57:25.471924 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349747    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 10:57:25.471924 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349749    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 10:57:25.471924 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349752    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 10:57:25.471924 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349754    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 10:57:25.471924 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349757    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 10:57:25.471924 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349759    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 10:57:25.471924 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349762    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 10:57:25.471924 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349764    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 10:57:25.471924 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349767    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 10:57:25.471924 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349769    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 10:57:25.472527 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349772    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 10:57:25.472527 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349775    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 10:57:25.472527 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349777    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 10:57:25.472527 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349780    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 10:57:25.472527 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349783    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 10:57:25.472527 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349785    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 10:57:25.472527 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349787    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 10:57:25.472527 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349790    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 10:57:25.472527 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349793    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 10:57:25.472527 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349795    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 10:57:25.472527 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349798    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 10:57:25.472527 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349800    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 10:57:25.472527 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349803    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 10:57:25.472527 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349805    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 10:57:25.472527 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349807    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 10:57:25.472527 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349813    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 10:57:25.472527 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349816    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 10:57:25.472527 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349818    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 10:57:25.472527 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349821    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 10:57:25.472527 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349824    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 10:57:25.473121 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349828    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 10:57:25.473121 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349831    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 10:57:25.473121 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.349833    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 10:57:25.473121 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350210    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 10:57:25.473121 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350215    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 10:57:25.473121 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350218    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 10:57:25.473121 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350221    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 10:57:25.473121 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350224    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 10:57:25.473121 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350227    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 10:57:25.473121 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350229    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 10:57:25.473121 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350234    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 10:57:25.473121 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350236    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 10:57:25.473121 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350239    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 10:57:25.473121 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350241    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 10:57:25.473121 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350244    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 10:57:25.473121 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350247    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 10:57:25.473121 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350249    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 10:57:25.473121 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350251    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 10:57:25.473121 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350254    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 10:57:25.473634 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350257    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 10:57:25.473634 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350259    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 10:57:25.473634 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350262    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 10:57:25.473634 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350264    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 10:57:25.473634 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350267    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 10:57:25.473634 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350269    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 10:57:25.473634 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350280    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 10:57:25.473634 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350284    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 10:57:25.473634 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350288    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 10:57:25.473634 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350291    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 10:57:25.473634 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350293    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 10:57:25.473634 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350296    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 10:57:25.473634 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350299    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 10:57:25.473634 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350301    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 10:57:25.473634 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350306    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 10:57:25.473634 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350327    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 10:57:25.473634 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350330    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 10:57:25.473634 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350333    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 10:57:25.473634 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350336    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 10:57:25.474117 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350339    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 10:57:25.474117 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350342    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 10:57:25.474117 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350355    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 10:57:25.474117 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350358    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 10:57:25.474117 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350362    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 10:57:25.474117 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350365    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 10:57:25.474117 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350367    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 10:57:25.474117 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350370    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 10:57:25.474117 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350373    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 10:57:25.474117 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350375    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 10:57:25.474117 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350378    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 10:57:25.474117 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350380    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 10:57:25.474117 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350383    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 10:57:25.474117 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350385    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 10:57:25.474117 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350388    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 10:57:25.474117 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350390    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 10:57:25.474117 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350392    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 10:57:25.474117 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350396    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 10:57:25.474117 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350399    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 10:57:25.474621 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350401    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 10:57:25.474621 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350403    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 10:57:25.474621 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350407    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 10:57:25.474621 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350410    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 10:57:25.474621 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350412    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 10:57:25.474621 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350415    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 10:57:25.474621 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350417    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 10:57:25.474621 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350420    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 10:57:25.474621 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350422    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 10:57:25.474621 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350425    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 10:57:25.474621 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350427    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 10:57:25.474621 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350430    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 10:57:25.474621 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350432    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 10:57:25.474621 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350435    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 10:57:25.474621 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350437    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 10:57:25.474621 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350440    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 10:57:25.474621 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350443    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 10:57:25.474621 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350448    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 10:57:25.474621 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350450    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 10:57:25.474621 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350453    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 10:57:25.475161 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350456    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 10:57:25.475161 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350459    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 10:57:25.475161 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350461    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 10:57:25.475161 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350463    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 10:57:25.475161 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350466    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 10:57:25.475161 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350468    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 10:57:25.475161 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350471    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 10:57:25.475161 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350473    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 10:57:25.475161 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350476    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 10:57:25.475161 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350478    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 10:57:25.475161 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350481    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 10:57:25.475161 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350483    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 10:57:25.475161 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350486    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 10:57:25.475161 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350488    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 10:57:25.475161 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350492    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 10:57:25.475161 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350494    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 10:57:25.475161 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350497    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 10:57:25.475161 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350499    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 10:57:25.475161 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350502    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 10:57:25.475161 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350504    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 10:57:25.475705 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350507    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 10:57:25.475705 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350509    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 10:57:25.475705 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350511    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 10:57:25.475705 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350514    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 10:57:25.475705 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.350516    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 10:57:25.475705 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350591    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 17 10:57:25.475705 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350601    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 17 10:57:25.475705 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350607    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 17 10:57:25.475705 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350612    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 17 10:57:25.475705 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350618    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 17 10:57:25.475705 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350621    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 17 10:57:25.475705 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350626    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 17 10:57:25.475705 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350630    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 17 10:57:25.475705 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350633    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 17 10:57:25.475705 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350636    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 17 10:57:25.475705 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350639    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 17 10:57:25.475705 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350642    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 17 10:57:25.475705 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350645    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 17 10:57:25.475705 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350648    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 17 10:57:25.475705 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350651    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 17 10:57:25.475705 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350654    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 17 10:57:25.475705 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350657    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 17 10:57:25.475705 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350661    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 17 10:57:25.475705 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350664    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 17 10:57:25.476393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350667    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 17 10:57:25.476393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350670    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 17 10:57:25.476393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350675    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 17 10:57:25.476393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350679    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 17 10:57:25.476393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350683    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 17 10:57:25.476393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350686    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 17 10:57:25.476393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350689    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 17 10:57:25.476393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350692    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 17 10:57:25.476393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350695    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 17 10:57:25.476393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350698    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 17 10:57:25.476393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350701    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 17 10:57:25.476393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350704    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 17 10:57:25.476393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350707    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 17 10:57:25.476393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350711    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 17 10:57:25.476393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350714    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 17 10:57:25.476393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350717    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 17 10:57:25.476393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350720    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 17 10:57:25.476393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350724    2563 flags.go:64] FLAG: --enable-server="true"
Jul 17 10:57:25.476393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350728    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 17 10:57:25.476393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350732    2563 flags.go:64] FLAG: --event-burst="100"
Jul 17 10:57:25.476393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350735    2563 flags.go:64] FLAG: --event-qps="50"
Jul 17 10:57:25.476393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350738    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 17 10:57:25.476393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350741    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 17 10:57:25.476393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350744    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 17 10:57:25.477130 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350747    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 17 10:57:25.477130 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350751    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 17 10:57:25.477130 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350754    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 17 10:57:25.477130 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350757    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 17 10:57:25.477130 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350760    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 17 10:57:25.477130 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350763    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 17 10:57:25.477130 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350765    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 17 10:57:25.477130 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350768    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 17 10:57:25.477130 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350771    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 17 10:57:25.477130 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350774    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 17 10:57:25.477130 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350778    2563 flags.go:64] FLAG: --feature-gates=""
Jul 17 10:57:25.477130 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350782    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 17 10:57:25.477130 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350785    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 17 10:57:25.477130 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350788    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 17 10:57:25.477130 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350791    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 17 10:57:25.477130 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350794    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 17 10:57:25.477130 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350797    2563 flags.go:64] FLAG: --help="false"
Jul 17 10:57:25.477130 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350800    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-140-233.ec2.internal"
Jul 17 10:57:25.477130 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350803    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 17 10:57:25.477130 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350807    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 17 10:57:25.477130 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350809    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 17 10:57:25.477130 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350813    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 17 10:57:25.477130 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350816    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 17 10:57:25.477765 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350819    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 17 10:57:25.477765 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350821    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 17 10:57:25.477765 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350825    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 17 10:57:25.477765 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350828    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 17 10:57:25.477765 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350833    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 17 10:57:25.477765 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350836    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 17 10:57:25.477765 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350839    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 17 10:57:25.477765 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350842    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 17 10:57:25.477765 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350845    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 17 10:57:25.477765 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350848    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 17 10:57:25.477765 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350851    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 17 10:57:25.477765 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350853    2563 flags.go:64] FLAG: --lock-file=""
Jul 17 10:57:25.477765 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350856    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 17 10:57:25.477765 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350859    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 17 10:57:25.477765 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350862    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 17 10:57:25.477765 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350867    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 17 10:57:25.477765 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350870    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 17 10:57:25.477765 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350873    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 17 10:57:25.477765 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350876    2563 flags.go:64] FLAG: --logging-format="text"
Jul 17 10:57:25.477765 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350880    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 17 10:57:25.477765 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350884    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 17 10:57:25.477765 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350886    2563 flags.go:64] FLAG: --manifest-url=""
Jul 17 10:57:25.477765 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350889    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 17 10:57:25.477765 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350894    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 17 10:57:25.477765 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350897    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 17 10:57:25.478485 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350901    2563 flags.go:64] FLAG: --max-pods="110"
Jul 17 10:57:25.478485 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350903    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 17 10:57:25.478485 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350906    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 17 10:57:25.478485 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350909    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 17 10:57:25.478485 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350912    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 17 10:57:25.478485 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350915    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 17 10:57:25.478485 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350918    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 17 10:57:25.478485 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350921    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 17 10:57:25.478485 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350929    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 17 10:57:25.478485 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350933    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 17 10:57:25.478485 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350936    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 17 10:57:25.478485 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350939    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 17 10:57:25.478485 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350943    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 17 10:57:25.478485 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350949    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 17 10:57:25.478485 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350951    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 17 10:57:25.478485 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350955    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 17 10:57:25.478485 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350958    2563 flags.go:64] FLAG: --port="10250"
Jul 17 10:57:25.478485 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350961    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 17 10:57:25.478485 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350964    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e77b43207f376bf0"
Jul 17 10:57:25.478485 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350967    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 17 10:57:25.478485 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350970    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 17 10:57:25.478485 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350973    2563 flags.go:64] FLAG: --register-node="true"
Jul 17 10:57:25.478485 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350976    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 17 10:57:25.478485 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350979    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 17 10:57:25.479230 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350982    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 17 10:57:25.479230 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350984    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 17 10:57:25.479230 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350989    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 17 10:57:25.479230 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350993    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 17 10:57:25.479230 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350996    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 17 10:57:25.479230 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.350999    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 17 10:57:25.479230 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351002    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 17 10:57:25.479230 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351004    2563 flags.go:64] FLAG: --runonce="false"
Jul 17 10:57:25.479230 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351007    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 17 10:57:25.479230 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351010    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 17 10:57:25.479230 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351013    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 17 10:57:25.479230 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351016    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 17 10:57:25.479230 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351019    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 17 10:57:25.479230 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351021    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 17 10:57:25.479230 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351024    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 17 10:57:25.479230 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351027    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 17 10:57:25.479230 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351030    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 17 10:57:25.479230 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351033    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 17 10:57:25.479230 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351037    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 17 10:57:25.479230 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351040    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 17 10:57:25.479230 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351043    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 17 10:57:25.479230 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351046    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 17 10:57:25.479230 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351049    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 17 10:57:25.479230 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351054    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 17 10:57:25.479230 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351057    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 17 10:57:25.479904 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351060    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 17 10:57:25.479904 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351078    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 17 10:57:25.479904 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351083    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 17 10:57:25.479904 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351085    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 17 10:57:25.479904 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351089    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 17 10:57:25.479904 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351092    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 17 10:57:25.479904 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351101    2563 flags.go:64] FLAG: --v="2"
Jul 17 10:57:25.479904 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351106    2563 flags.go:64] FLAG: --version="false"
Jul 17 10:57:25.479904 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351109    2563 flags.go:64] FLAG: --vmodule=""
Jul 17 10:57:25.479904 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351114    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 17 10:57:25.479904 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.351118    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 17 10:57:25.479904 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351232    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 10:57:25.479904 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351237    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 10:57:25.479904 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351240    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 10:57:25.479904 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351244    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 10:57:25.479904 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351246    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 10:57:25.479904 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351249    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 10:57:25.479904 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351252    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 10:57:25.479904 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351255    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 10:57:25.479904 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351258    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 10:57:25.479904 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351261    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 10:57:25.479904 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351263    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 10:57:25.480556 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351266    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 10:57:25.480556 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351268    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 10:57:25.480556 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351271    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 10:57:25.480556 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351275    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 10:57:25.480556 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351277    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 10:57:25.480556 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351280    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 10:57:25.480556 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351282    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 10:57:25.480556 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351286    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 10:57:25.480556 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351288    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 10:57:25.480556 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351291    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 10:57:25.480556 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351293    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 10:57:25.480556 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351296    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 10:57:25.480556 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351298    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 10:57:25.480556 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351301    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 10:57:25.480556 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351303    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 10:57:25.480556 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351306    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 10:57:25.480556 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351309    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 10:57:25.480556 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351312    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 10:57:25.480556 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351314    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 10:57:25.480556 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351318    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 10:57:25.481084 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351320    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 10:57:25.481084 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351323    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 10:57:25.481084 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351325    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 10:57:25.481084 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351328    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 10:57:25.481084 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351330    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 10:57:25.481084 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351333    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 10:57:25.481084 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351335    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 10:57:25.481084 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351338    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 10:57:25.481084 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351341    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 10:57:25.481084 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351344    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 10:57:25.481084 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351346    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 10:57:25.481084 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351348    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 10:57:25.481084 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351351    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 10:57:25.481084 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351353    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 10:57:25.481084 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351356    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 10:57:25.481084 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351361    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 10:57:25.481084 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351365    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 10:57:25.481084 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351368    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 10:57:25.481084 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351370    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 10:57:25.481608 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351373    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 10:57:25.481608 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351376    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 10:57:25.481608 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351378    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 10:57:25.481608 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351381    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 10:57:25.481608 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351383    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 10:57:25.481608 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351385    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 10:57:25.481608 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351388    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 10:57:25.481608 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351390    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 10:57:25.481608 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351393    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 10:57:25.481608 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351395    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 10:57:25.481608 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351398    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 10:57:25.481608 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351400    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 10:57:25.481608 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351404    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 10:57:25.481608 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351406    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 10:57:25.481608 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351409    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 10:57:25.481608 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351411    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 10:57:25.481608 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351413    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 10:57:25.481608 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351416    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 10:57:25.481608 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351418    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 10:57:25.482124 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351421    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 10:57:25.482124 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351423    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 10:57:25.482124 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351426    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 10:57:25.482124 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351428    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 10:57:25.482124 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351431    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 10:57:25.482124 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351433    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 10:57:25.482124 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351436    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 10:57:25.482124 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351438    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 10:57:25.482124 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351441    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 10:57:25.482124 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351445    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 10:57:25.482124 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351447    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 10:57:25.482124 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351450    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 10:57:25.482124 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351452    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 10:57:25.482124 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351454    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 10:57:25.482124 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351458    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 10:57:25.482124 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351462    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 10:57:25.482124 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351465    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 10:57:25.482124 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351468    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 10:57:25.482124 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351471    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 10:57:25.482635 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351474    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 10:57:25.482635 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351476    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 10:57:25.482635 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351479    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 10:57:25.482635 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351481    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 10:57:25.482635 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351483    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 10:57:25.482635 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351486    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 10:57:25.482635 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351490    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 10:57:25.482635 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351492    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 10:57:25.482635 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351495    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 10:57:25.482635 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351497    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 10:57:25.482635 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.351500    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 10:57:25.482635 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.352088    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 10:57:25.482635 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.358950    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 17 10:57:25.482635 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.358967    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 17 10:57:25.482635 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.359648    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 17 10:57:25.483040 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.359655    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 17 10:57:25.483040 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359713    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 10:57:25.483040 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359719    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 10:57:25.483040 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359721    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 10:57:25.483040 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359725    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 10:57:25.483040 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359727    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 10:57:25.483040 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359730    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 10:57:25.483040 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359734    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 10:57:25.483040 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359736    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 10:57:25.483040 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359739    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 10:57:25.483040 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359742    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 10:57:25.483040 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359744    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 10:57:25.483040 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359747    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 10:57:25.483040 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359751    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 10:57:25.483040 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359754    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 10:57:25.483040 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359757    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 10:57:25.483040 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359759    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 10:57:25.483040 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359762    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 10:57:25.483040 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359766    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 10:57:25.483563 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359769    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 10:57:25.483563 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359772    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 10:57:25.483563 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359774    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 10:57:25.483563 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359777    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 10:57:25.483563 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359779    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 10:57:25.483563 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359782    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 10:57:25.483563 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359784    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 10:57:25.483563 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359787    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 10:57:25.483563 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359789    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 10:57:25.483563 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359792    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 10:57:25.483563 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359799    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 10:57:25.483563 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359802    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 10:57:25.483563 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359806    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 10:57:25.483563 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359810    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 10:57:25.483563 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359813    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 10:57:25.483563 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359816    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 10:57:25.483563 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359818    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 10:57:25.483563 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359821    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 10:57:25.483563 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359823    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 10:57:25.484036 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359826    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 10:57:25.484036 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359828    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 10:57:25.484036 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359830    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 10:57:25.484036 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359833    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 10:57:25.484036 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359836    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 10:57:25.484036 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359838    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 10:57:25.484036 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359841    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 10:57:25.484036 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359843    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 10:57:25.484036 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359847    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 10:57:25.484036 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359849    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 10:57:25.484036 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359852    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 10:57:25.484036 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359854    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 10:57:25.484036 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359857    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 10:57:25.484036 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359859    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 10:57:25.484036 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359862    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 10:57:25.484036 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359865    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 10:57:25.484036 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359867    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 10:57:25.484036 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359870    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 10:57:25.484036 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359872    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 10:57:25.484036 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359875    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 10:57:25.484580 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359877    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 10:57:25.484580 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359879    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 10:57:25.484580 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359882    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 10:57:25.484580 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359887    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 10:57:25.484580 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359889    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 10:57:25.484580 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359892    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 10:57:25.484580 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359895    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 10:57:25.484580 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359898    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 10:57:25.484580 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359900    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 10:57:25.484580 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359902    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 10:57:25.484580 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359905    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 10:57:25.484580 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359907    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 10:57:25.484580 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359910    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 10:57:25.484580 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359912    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 10:57:25.484580 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359914    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 10:57:25.484580 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359917    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 10:57:25.484580 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359919    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 10:57:25.484580 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359922    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 10:57:25.484580 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359924    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 10:57:25.484580 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359927    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 10:57:25.485101 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359930    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 10:57:25.485101 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359935    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 10:57:25.485101 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359937    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 10:57:25.485101 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359941    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 10:57:25.485101 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359944    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 10:57:25.485101 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359946    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 10:57:25.485101 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359949    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 10:57:25.485101 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359952    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 10:57:25.485101 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359954    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 10:57:25.485101 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359957    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 10:57:25.485101 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359959    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 10:57:25.485101 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359961    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 10:57:25.485101 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359964    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 10:57:25.485101 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359966    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 10:57:25.485101 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359969    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 10:57:25.485101 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359977    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 10:57:25.485101 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359979    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 10:57:25.485101 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359982    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 10:57:25.485101 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359984    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 10:57:25.485101 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359987    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 10:57:25.485757 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359989    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 10:57:25.485757 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.359992    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 10:57:25.485757 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.359997    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 10:57:25.485757 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360128    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 10:57:25.485757 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360133    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 10:57:25.485757 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360136    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 10:57:25.485757 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360139    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 10:57:25.485757 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360142    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 10:57:25.485757 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360145    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 10:57:25.485757 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360147    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 10:57:25.485757 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360150    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 10:57:25.485757 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360152    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 10:57:25.485757 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360155    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 10:57:25.485757 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360157    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 10:57:25.485757 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360160    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 10:57:25.486178 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360163    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 10:57:25.486178 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360165    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 10:57:25.486178 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360167    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 10:57:25.486178 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360170    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 10:57:25.486178 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360172    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 10:57:25.486178 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360175    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 10:57:25.486178 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360177    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 10:57:25.486178 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360180    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 10:57:25.486178 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360182    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 10:57:25.486178 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360185    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 10:57:25.486178 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360188    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 10:57:25.486178 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360190    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 10:57:25.486178 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360194    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 10:57:25.486178 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360198    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 10:57:25.486178 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360200    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 10:57:25.486178 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360203    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 10:57:25.486178 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360205    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 10:57:25.486178 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360208    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 10:57:25.486178 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360210    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 10:57:25.486178 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360213    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 10:57:25.486724 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360215    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 10:57:25.486724 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360217    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 10:57:25.486724 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360220    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 10:57:25.486724 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360222    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 10:57:25.486724 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360226    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 10:57:25.486724 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360229    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 10:57:25.486724 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360232    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 10:57:25.486724 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360235    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 10:57:25.486724 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360238    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 10:57:25.486724 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360240    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 10:57:25.486724 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360243    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 10:57:25.486724 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360246    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 10:57:25.486724 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360249    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 10:57:25.486724 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360252    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 10:57:25.486724 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360256    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 10:57:25.486724 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360259    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 10:57:25.486724 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360261    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 10:57:25.486724 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360264    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 10:57:25.487228 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360267    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 10:57:25.487228 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360270    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 10:57:25.487228 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360273    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 10:57:25.487228 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360276    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 10:57:25.487228 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360278    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 10:57:25.487228 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360281    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 10:57:25.487228 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360285    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 10:57:25.487228 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360289    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 10:57:25.487228 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360292    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 10:57:25.487228 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360295    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 10:57:25.487228 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360298    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 10:57:25.487228 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360301    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 10:57:25.487228 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360304    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 10:57:25.487228 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360325    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 10:57:25.487228 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360328    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 10:57:25.487228 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360331    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 10:57:25.487228 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360334    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 10:57:25.487228 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360336    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 10:57:25.487228 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360339    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 10:57:25.487228 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360342    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 10:57:25.487749 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360344    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 10:57:25.487749 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360347    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 10:57:25.487749 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360350    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 10:57:25.487749 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360352    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 10:57:25.487749 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360355    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 10:57:25.487749 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360357    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 10:57:25.487749 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360360    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 10:57:25.487749 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360363    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 10:57:25.487749 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360366    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 10:57:25.487749 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360368    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 10:57:25.487749 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360370    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 10:57:25.487749 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360373    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 10:57:25.487749 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360375    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 10:57:25.487749 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360378    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 10:57:25.487749 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360381    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 10:57:25.487749 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360383    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 10:57:25.487749 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360385    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 10:57:25.487749 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360388    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 10:57:25.487749 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360391    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 10:57:25.487749 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360395    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 10:57:25.488285 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360398    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 10:57:25.488285 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360401    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 10:57:25.488285 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360403    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 10:57:25.488285 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360406    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 10:57:25.488285 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360409    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 10:57:25.488285 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360411    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 10:57:25.488285 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360414    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 10:57:25.488285 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360416    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 10:57:25.488285 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:25.360418    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 10:57:25.488285 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.360425    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 10:57:25.488285 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.360540    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 17 10:57:25.488285 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.362730    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 17 10:57:25.488285 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.364138    2563 server.go:1019] "Starting client certificate rotation"
Jul 17 10:57:25.488285 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.364250    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 10:57:25.488647 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.364311    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 10:57:25.488647 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.367000    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 10:57:25.488647 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.372429    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 10:57:25.488647 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.387271    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 17 10:57:25.488647 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.389336    2563 log.go:25] "Validated CRI v1 image API"
Jul 17 10:57:25.488647 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.390156    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 17 10:57:25.488647 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.395122    2563 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d7efa3a9-22d1-447c-99a7-2ff10cf438b5:/dev/nvme0n1p3 e1c3ce31-5852-418e-99ad-3a3333d4cbc3:/dev/nvme0n1p4]
Jul 17 10:57:25.488647 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.395152    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 17 10:57:25.488886 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.400489    2563 manager.go:216] Machine: {Timestamp:2026-07-17 10:57:25.399306776 +0000 UTC m=+0.328846408 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109937 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26147c6e3a6982d7489a18bea0fc86 SystemUUID:ec26147c-6e3a-6982-d748-9a18bea0fc86 BootID:7b65df99-61a1-4862-ae9e-c25fae9be705 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5a:4d:71:67:5b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5a:4d:71:67:5b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:ef:ec:6f:d4:86 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 17 10:57:25.488886 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.400598    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 17 10:57:25.488886 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.400710    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 17 10:57:25.488886 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.403695    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 17 10:57:25.488886 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.403722    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-233.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 17 10:57:25.488886 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.404392    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 17 10:57:25.488886 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.404402    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 17 10:57:25.488886 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.404410    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 10:57:25.488886 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.405006    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 10:57:25.488886 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.406331    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 17 10:57:25.488886 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.409641    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 10:57:25.488886 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.409746    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 17 10:57:25.488886 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.409771    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 17 10:57:25.488886 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.409781    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 17 10:57:25.488886 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.409796    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 17 10:57:25.488886 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.409807    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 17 10:57:25.488886 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.409815    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 17 10:57:25.488886 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.410154    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.410171    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.412269    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.412577    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.414091    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.414122    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.415204    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.415218    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.415224    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.415230    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.415238    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.415244    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.415255    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.415260    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.415266    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.415271    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.415286    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.415292    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.416098    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.416108    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.419721    2563 server.go:1268] "Started kubelet"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.420274    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.421029    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.421125    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.421979    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-233.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:25.422084    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-233.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:25.422148    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.423101    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.423436    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:25.427403    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.435335    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.435350    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:25.435431    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-233.ec2.internal\" not found"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.436407    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.436884    2563 factory.go:55] Registering systemd factory
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.436902    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.437372    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.437390    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.437399    2563 factory.go:153] Registering CRI-O factory
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.437414    2563 factory.go:223] Registration of the crio container factory successfully
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.437467    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.437493    2563 factory.go:103] Registering Raw factory
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.437510    2563 manager.go:1196] Started watching for new ooms in manager
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.437524    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.437531    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.438058    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vsps7"
Jul 17 10:57:25.489407 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.438520    2563 manager.go:349] Starting recovery of all containers
Jul 17 10:57:25.490653 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:25.448397    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-233.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 17 10:57:25.490653 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.448419    2563 manager.go:354] Recovery completed
Jul 17 10:57:25.490653 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:25.448469    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 17 10:57:25.490653 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:25.448383    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-233.ec2.internal.18c30e85d9fca238  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-233.ec2.internal,UID:ip-10-0-140-233.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-233.ec2.internal,},FirstTimestamp:2026-07-17 10:57:25.419500088 +0000 UTC m=+0.349039731,LastTimestamp:2026-07-17 10:57:25.419500088 +0000 UTC m=+0.349039731,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-233.ec2.internal,}"
Jul 17 10:57:25.490653 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.452709    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 10:57:25.490653 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.454753    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 10:57:25.490653 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.454785    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 10:57:25.490653 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.454800    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-233.ec2.internal" event="NodeHasSufficientPID"
Jul 17 10:57:25.490653 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.455294    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vsps7"
Jul 17 10:57:25.490653 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.455453    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 17 10:57:25.490653 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.455466    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 17 10:57:25.490653 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.455486    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 10:57:25.490653 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.455491    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 10:57:25.490653 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.456180    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 10:57:25.490653 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.456207    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 10:57:25.490653 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.456221    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-233.ec2.internal" event="NodeHasSufficientPID"
Jul 17 10:57:25.490653 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.457367    2563 policy_none.go:49] "None policy: Start"
Jul 17 10:57:25.490653 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.457389    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 17 10:57:25.490653 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.457400    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 17 10:57:25.490653 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.458565    2563 policy_none.go:47] "Start"
Jul 17 10:57:25.535658 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:25.535623    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-233.ec2.internal\" not found"
Jul 17 10:57:25.552140 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.551972    2563 manager.go:337] "Starting Device Plugin manager"
Jul 17 10:57:25.552281 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:25.552191    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 17 10:57:25.552281 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.552208    2563 server.go:87] "Starting device plugin registration server"
Jul 17 10:57:25.552539 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.552525    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 17 10:57:25.552594 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.552543    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 17 10:57:25.552642 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.552623    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 17 10:57:25.552779 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.552712    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 17 10:57:25.552779 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.552728    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 17 10:57:25.553284 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:25.553260    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 17 10:57:25.553360 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:25.553310    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-233.ec2.internal\" not found"
Jul 17 10:57:25.652933 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.652902    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 10:57:25.654150 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.653966    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 10:57:25.654150 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.654001    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 10:57:25.654150 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.654017    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-233.ec2.internal" event="NodeHasSufficientPID"
Jul 17 10:57:25.654150 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.654054    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-233.ec2.internal"
Jul 17 10:57:25.656697 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:25.656677    2563 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-233.ec2.internal\" not found" node="ip-10-0-140-233.ec2.internal"
Jul 17 10:57:25.657699 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.657675    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 17 10:57:25.659071 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.659036    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 17 10:57:25.659178 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.659094    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 17 10:57:25.659178 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.659119    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 17 10:57:25.659178 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:25.659168    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 17 10:57:25.661520 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.661492    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 10:57:25.663484 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.663468    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-233.ec2.internal"
Jul 17 10:57:25.663562 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:25.663492    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-233.ec2.internal\": node \"ip-10-0-140-233.ec2.internal\" not found"
Jul 17 10:57:25.714485 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:25.714451    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-233.ec2.internal\" not found"
Jul 17 10:57:25.760578 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.760494    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-233.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-233.ec2.internal"]
Jul 17 10:57:25.760704 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.760612    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 10:57:25.762451 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.762432    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 10:57:25.762561 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.762468    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 10:57:25.762561 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.762483    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-233.ec2.internal" event="NodeHasSufficientPID"
Jul 17 10:57:25.763957 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.763939    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 10:57:25.764075 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.764060    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-233.ec2.internal"
Jul 17 10:57:25.764135 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.764115    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 10:57:25.765225 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.765185    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 10:57:25.765309 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.765230    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 10:57:25.765309 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.765241    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-233.ec2.internal" event="NodeHasSufficientPID"
Jul 17 10:57:25.765309 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.765185    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 10:57:25.765309 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.765309    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 10:57:25.765441 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.765325    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-233.ec2.internal" event="NodeHasSufficientPID"
Jul 17 10:57:25.766782 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.766763    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-233.ec2.internal"
Jul 17 10:57:25.766876 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.766797    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 10:57:25.768226 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.768208    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 10:57:25.768331 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.768243    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 10:57:25.768331 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.768257    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-233.ec2.internal" event="NodeHasSufficientPID"
Jul 17 10:57:25.788090 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:25.788056    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-233.ec2.internal\" not found" node="ip-10-0-140-233.ec2.internal"
Jul 17 10:57:25.792912 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:25.792892    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-233.ec2.internal\" not found" node="ip-10-0-140-233.ec2.internal"
Jul 17 10:57:25.815515 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:25.815483    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-233.ec2.internal\" not found"
Jul 17 10:57:25.839023 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.838991    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2c42ae2a6854f18bdedf80ad478b60e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-233.ec2.internal\" (UID: \"d2c42ae2a6854f18bdedf80ad478b60e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-233.ec2.internal"
Jul 17 10:57:25.839023 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.839025    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/098d9536b1eb81f9184c22a2bb857164-config\") pod \"kube-apiserver-proxy-ip-10-0-140-233.ec2.internal\" (UID: \"098d9536b1eb81f9184c22a2bb857164\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-233.ec2.internal"
Jul 17 10:57:25.839201 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.839043    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d2c42ae2a6854f18bdedf80ad478b60e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-233.ec2.internal\" (UID: \"d2c42ae2a6854f18bdedf80ad478b60e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-233.ec2.internal"
Jul 17 10:57:25.916024 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:25.915997    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-233.ec2.internal\" not found"
Jul 17 10:57:25.940217 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.940188    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d2c42ae2a6854f18bdedf80ad478b60e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-233.ec2.internal\" (UID: \"d2c42ae2a6854f18bdedf80ad478b60e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-233.ec2.internal"
Jul 17 10:57:25.940330 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.940248    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d2c42ae2a6854f18bdedf80ad478b60e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-233.ec2.internal\" (UID: \"d2c42ae2a6854f18bdedf80ad478b60e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-233.ec2.internal"
Jul 17 10:57:25.940330 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.940302    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2c42ae2a6854f18bdedf80ad478b60e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-233.ec2.internal\" (UID: \"d2c42ae2a6854f18bdedf80ad478b60e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-233.ec2.internal"
Jul 17 10:57:25.940441 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.940335    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/098d9536b1eb81f9184c22a2bb857164-config\") pod \"kube-apiserver-proxy-ip-10-0-140-233.ec2.internal\" (UID: \"098d9536b1eb81f9184c22a2bb857164\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-233.ec2.internal"
Jul 17 10:57:25.940441 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.940392    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/098d9536b1eb81f9184c22a2bb857164-config\") pod \"kube-apiserver-proxy-ip-10-0-140-233.ec2.internal\" (UID: \"098d9536b1eb81f9184c22a2bb857164\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-233.ec2.internal"
Jul 17 10:57:25.940441 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:25.940410    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2c42ae2a6854f18bdedf80ad478b60e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-233.ec2.internal\" (UID: \"d2c42ae2a6854f18bdedf80ad478b60e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-233.ec2.internal"
Jul 17 10:57:26.018421 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:26.018337    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-233.ec2.internal\" not found"
Jul 17 10:57:26.094656 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.094618    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-233.ec2.internal"
Jul 17 10:57:26.096989 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.096955    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-233.ec2.internal"
Jul 17 10:57:26.118578 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:26.118543    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-233.ec2.internal\" not found"
Jul 17 10:57:26.220184 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:26.220132    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-233.ec2.internal\" not found"
Jul 17 10:57:26.320351 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:26.320262    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-233.ec2.internal\" not found"
Jul 17 10:57:26.349027 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.349003    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 10:57:26.363646 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.363615    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 17 10:57:26.363775 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.363755    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 10:57:26.363850 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.363819    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 10:57:26.410891 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.410855    2563 apiserver.go:52] "Watching apiserver"
Jul 17 10:57:26.431374 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.430518    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 17 10:57:26.431580 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.431536    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-qqqf9","openshift-multus/multus-additional-cni-plugins-pcg7j","openshift-multus/network-metrics-daemon-s5c9b","kube-system/konnectivity-agent-kz6gd","openshift-multus/multus-j4mp9","openshift-network-diagnostics/network-check-target-g5hnv","openshift-network-operator/iptables-alerter-pvvr8","openshift-ovn-kubernetes/ovnkube-node-nj5mj","kube-system/global-pull-secret-syncer-b7fx4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59","openshift-cluster-node-tuning-operator/tuned-l426v","openshift-dns/node-resolver-87r57"]
Jul 17 10:57:26.433371 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.433350    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qqqf9"
Jul 17 10:57:26.434495 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.434477    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pcg7j"
Jul 17 10:57:26.435623 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.435595    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-233.ec2.internal"
Jul 17 10:57:26.435623 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.435614    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 17 10:57:26.435901 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.435877    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 17 10:57:26.435901 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.435891    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 17 10:57:26.436106 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.435922    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hkhm8\""
Jul 17 10:57:26.436106 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.435934    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:26.436106 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.436041    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 17 10:57:26.436106 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:26.436037    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c9b" podUID="e9c3cb80-365e-4c10-bde0-1e9f17948a97"
Jul 17 10:57:26.436861 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.436842    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 17 10:57:26.436977 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.436879    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 17 10:57:26.436977 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.436885    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 17 10:57:26.437083 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.437018    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vw8c9\""
Jul 17 10:57:26.437083 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.437012    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 17 10:57:26.437083 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.437043    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 17 10:57:26.437347 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.437331    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kz6gd"
Jul 17 10:57:26.438635 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.438618    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.439862 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.439844    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:57:26.439959 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:26.439916    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5hnv" podUID="f2830f09-58f3-425c-a831-e5e34b542e78"
Jul 17 10:57:26.439959 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.439939    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7pvpj\""
Jul 17 10:57:26.440076 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.440027    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 17 10:57:26.440214 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.440198    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 17 10:57:26.440897 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.440880    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 17 10:57:26.441005 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.440985    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-psgt2\""
Jul 17 10:57:26.441146 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.441125    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pvvr8"
Jul 17 10:57:26.442669 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.442654    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.443313 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.443292    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9c3cb80-365e-4c10-bde0-1e9f17948a97-metrics-certs\") pod \"network-metrics-daemon-s5c9b\" (UID: \"e9c3cb80-365e-4c10-bde0-1e9f17948a97\") " pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:26.443416 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.443318    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73402c16-36b4-442c-9009-4d41bd9003fc-konnectivity-ca\") pod \"konnectivity-agent-kz6gd\" (UID: \"73402c16-36b4-442c-9009-4d41bd9003fc\") " pod="kube-system/konnectivity-agent-kz6gd"
Jul 17 10:57:26.443416 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.443342    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lgbt\" (UniqueName: \"kubernetes.io/projected/72f5631e-c70d-47b8-9405-d8b986fd12e7-kube-api-access-8lgbt\") pod \"node-ca-qqqf9\" (UID: \"72f5631e-c70d-47b8-9405-d8b986fd12e7\") " pod="openshift-image-registry/node-ca-qqqf9"
Jul 17 10:57:26.443416 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.443381    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b3498a1-e561-4239-8878-ad9595ad312a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pcg7j\" (UID: \"4b3498a1-e561-4239-8878-ad9595ad312a\") " pod="openshift-multus/multus-additional-cni-plugins-pcg7j"
Jul 17 10:57:26.443416 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.443410    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b3498a1-e561-4239-8878-ad9595ad312a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pcg7j\" (UID: \"4b3498a1-e561-4239-8878-ad9595ad312a\") " pod="openshift-multus/multus-additional-cni-plugins-pcg7j"
Jul 17 10:57:26.443580 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.443447    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxbqs\" (UniqueName: \"kubernetes.io/projected/e9c3cb80-365e-4c10-bde0-1e9f17948a97-kube-api-access-cxbqs\") pod \"network-metrics-daemon-s5c9b\" (UID: \"e9c3cb80-365e-4c10-bde0-1e9f17948a97\") " pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:26.443580 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.443458    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-x6s7k\""
Jul 17 10:57:26.443580 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.443553    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/72f5631e-c70d-47b8-9405-d8b986fd12e7-serviceca\") pod \"node-ca-qqqf9\" (UID: \"72f5631e-c70d-47b8-9405-d8b986fd12e7\") " pod="openshift-image-registry/node-ca-qqqf9"
Jul 17 10:57:26.443723 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.443614    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b3498a1-e561-4239-8878-ad9595ad312a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pcg7j\" (UID: \"4b3498a1-e561-4239-8878-ad9595ad312a\") " pod="openshift-multus/multus-additional-cni-plugins-pcg7j"
Jul 17 10:57:26.443723 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.443674    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73402c16-36b4-442c-9009-4d41bd9003fc-agent-certs\") pod \"konnectivity-agent-kz6gd\" (UID: \"73402c16-36b4-442c-9009-4d41bd9003fc\") " pod="kube-system/konnectivity-agent-kz6gd"
Jul 17 10:57:26.443723 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.443687    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 17 10:57:26.443723 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.443708    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b3498a1-e561-4239-8878-ad9595ad312a-cni-binary-copy\") pod \"multus-additional-cni-plugins-pcg7j\" (UID: \"4b3498a1-e561-4239-8878-ad9595ad312a\") " pod="openshift-multus/multus-additional-cni-plugins-pcg7j"
Jul 17 10:57:26.443936 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.443725    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 17 10:57:26.443936 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.443740    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swx4c\" (UniqueName: \"kubernetes.io/projected/4b3498a1-e561-4239-8878-ad9595ad312a-kube-api-access-swx4c\") pod \"multus-additional-cni-plugins-pcg7j\" (UID: \"4b3498a1-e561-4239-8878-ad9595ad312a\") " pod="openshift-multus/multus-additional-cni-plugins-pcg7j"
Jul 17 10:57:26.443936 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.443764    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72f5631e-c70d-47b8-9405-d8b986fd12e7-host\") pod \"node-ca-qqqf9\" (UID: \"72f5631e-c70d-47b8-9405-d8b986fd12e7\") " pod="openshift-image-registry/node-ca-qqqf9"
Jul 17 10:57:26.443936 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.443788    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b3498a1-e561-4239-8878-ad9595ad312a-system-cni-dir\") pod \"multus-additional-cni-plugins-pcg7j\" (UID: \"4b3498a1-e561-4239-8878-ad9595ad312a\") " pod="openshift-multus/multus-additional-cni-plugins-pcg7j"
Jul 17 10:57:26.443936 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.443811    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b3498a1-e561-4239-8878-ad9595ad312a-cnibin\") pod \"multus-additional-cni-plugins-pcg7j\" (UID: \"4b3498a1-e561-4239-8878-ad9595ad312a\") " pod="openshift-multus/multus-additional-cni-plugins-pcg7j"
Jul 17 10:57:26.443936 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.443828    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4b3498a1-e561-4239-8878-ad9595ad312a-os-release\") pod \"multus-additional-cni-plugins-pcg7j\" (UID: \"4b3498a1-e561-4239-8878-ad9595ad312a\") " pod="openshift-multus/multus-additional-cni-plugins-pcg7j"
Jul 17 10:57:26.443936 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.443829    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 17 10:57:26.444174 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.443973    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:26.444174 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:26.444027    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7fx4" podUID="84ae7bd0-6126-491f-9126-2d64506af889"
Jul 17 10:57:26.445487 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.445474    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59"
Jul 17 10:57:26.446080 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.446062    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4nrpq\""
Jul 17 10:57:26.446277 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.446251    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 17 10:57:26.446277 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.446278    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 17 10:57:26.446443 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.446287    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 17 10:57:26.446443 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.446303    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 17 10:57:26.446586 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.446571    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 17 10:57:26.446704 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.446680    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 17 10:57:26.446959 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.446945    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.447761 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.447748    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 17 10:57:26.447911 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.447897    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 17 10:57:26.448005 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.447990    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 17 10:57:26.448055 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.448041    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-87k2x\""
Jul 17 10:57:26.448406 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.448384    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-87r57"
Jul 17 10:57:26.449357 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.449300    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 17 10:57:26.449357 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.449300    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g78kz\""
Jul 17 10:57:26.451569 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.449524    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 17 10:57:26.451820 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.451796    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 17 10:57:26.451923 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.451820    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 17 10:57:26.451923 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.451884    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fdx6n\""
Jul 17 10:57:26.452788 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.452772    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 10:57:26.453033 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.452956    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-233.ec2.internal"
Jul 17 10:57:26.453197 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.453145    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-233.ec2.internal"]
Jul 17 10:57:26.457484 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.457465    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 10:57:26.459451 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.459402    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 10:52:25 +0000 UTC" deadline="2028-03-22 04:36:41.888386508 +0000 UTC"
Jul 17 10:57:26.459514 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.459450    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14729h39m15.428940675s"
Jul 17 10:57:26.463938 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.463920    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 10:57:26.464204 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.464184    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-233.ec2.internal"]
Jul 17 10:57:26.504974 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.504950    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fzfvt"
Jul 17 10:57:26.514960 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.514939    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fzfvt"
Jul 17 10:57:26.538118 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.538095    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 17 10:57:26.544022 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.543998    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/72f5631e-c70d-47b8-9405-d8b986fd12e7-serviceca\") pod \"node-ca-qqqf9\" (UID: \"72f5631e-c70d-47b8-9405-d8b986fd12e7\") " pod="openshift-image-registry/node-ca-qqqf9"
Jul 17 10:57:26.544180 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544033    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-etc-systemd\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.544180 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544052    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/609409ce-006b-440e-8403-dda1bef2dd5b-tmp\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.544180 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544081    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f1f509b8-7e29-445b-ad25-dd54a1c6dc07-hosts-file\") pod \"node-resolver-87r57\" (UID: \"f1f509b8-7e29-445b-ad25-dd54a1c6dc07\") " pod="openshift-dns/node-resolver-87r57"
Jul 17 10:57:26.544180 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544129    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-host-var-lib-kubelet\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.544401 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544249    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-run-systemd\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.544401 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544267    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/55d1de37-8635-4de8-ab59-4ebfd0346a40-ovn-node-metrics-cert\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.544401 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swx4c\" (UniqueName: \"kubernetes.io/projected/4b3498a1-e561-4239-8878-ad9595ad312a-kube-api-access-swx4c\") pod \"multus-additional-cni-plugins-pcg7j\" (UID: \"4b3498a1-e561-4239-8878-ad9595ad312a\") " pod="openshift-multus/multus-additional-cni-plugins-pcg7j"
Jul 17 10:57:26.544401 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-etc-kubernetes\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.544401 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544338    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-lib-modules\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.544401 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544360    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-host\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.544401 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544383    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/609409ce-006b-440e-8403-dda1bef2dd5b-etc-tuned\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.544737 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544427    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-etc-kubernetes\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.544737 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544464    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-run-ovn\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.544737 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544492    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fc69e11f-7c22-4067-8169-886d341a4b70-socket-dir\") pod \"aws-ebs-csi-driver-node-p9h59\" (UID: \"fc69e11f-7c22-4067-8169-886d341a4b70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59"
Jul 17 10:57:26.544737 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544524    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b3498a1-e561-4239-8878-ad9595ad312a-system-cni-dir\") pod \"multus-additional-cni-plugins-pcg7j\" (UID: \"4b3498a1-e561-4239-8878-ad9595ad312a\") " pod="openshift-multus/multus-additional-cni-plugins-pcg7j"
Jul 17 10:57:26.544737 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544551    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73402c16-36b4-442c-9009-4d41bd9003fc-konnectivity-ca\") pod \"konnectivity-agent-kz6gd\" (UID: \"73402c16-36b4-442c-9009-4d41bd9003fc\") " pod="kube-system/konnectivity-agent-kz6gd"
Jul 17 10:57:26.544737 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544554    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/72f5631e-c70d-47b8-9405-d8b986fd12e7-serviceca\") pod \"node-ca-qqqf9\" (UID: \"72f5631e-c70d-47b8-9405-d8b986fd12e7\") " pod="openshift-image-registry/node-ca-qqqf9"
Jul 17 10:57:26.544737 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544577    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9v8m\" (UniqueName: \"kubernetes.io/projected/f2830f09-58f3-425c-a831-e5e34b542e78-kube-api-access-n9v8m\") pod \"network-check-target-g5hnv\" (UID: \"f2830f09-58f3-425c-a831-e5e34b542e78\") " pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:57:26.544737 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544607    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b3498a1-e561-4239-8878-ad9595ad312a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pcg7j\" (UID: \"4b3498a1-e561-4239-8878-ad9595ad312a\") " pod="openshift-multus/multus-additional-cni-plugins-pcg7j"
Jul 17 10:57:26.544737 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544618    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b3498a1-e561-4239-8878-ad9595ad312a-system-cni-dir\") pod \"multus-additional-cni-plugins-pcg7j\" (UID: \"4b3498a1-e561-4239-8878-ad9595ad312a\") " pod="openshift-multus/multus-additional-cni-plugins-pcg7j"
Jul 17 10:57:26.544737 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544635    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxbqs\" (UniqueName: \"kubernetes.io/projected/e9c3cb80-365e-4c10-bde0-1e9f17948a97-kube-api-access-cxbqs\") pod \"network-metrics-daemon-s5c9b\" (UID: \"e9c3cb80-365e-4c10-bde0-1e9f17948a97\") " pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:26.544737 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544659    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-run\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.544737 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544683    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-hostroot\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.544737 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544708    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r76hn\" (UniqueName: \"kubernetes.io/projected/d2304352-6b6e-42d3-af75-146ba8b85e0b-kube-api-access-r76hn\") pod \"iptables-alerter-pvvr8\" (UID: \"d2304352-6b6e-42d3-af75-146ba8b85e0b\") " pod="openshift-network-operator/iptables-alerter-pvvr8"
Jul 17 10:57:26.544737 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544731    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-host-slash\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.545380 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544763    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfh6w\" (UniqueName: \"kubernetes.io/projected/55d1de37-8635-4de8-ab59-4ebfd0346a40-kube-api-access-cfh6w\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.545380 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544790    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcrfj\" (UniqueName: \"kubernetes.io/projected/f1f509b8-7e29-445b-ad25-dd54a1c6dc07-kube-api-access-mcrfj\") pod \"node-resolver-87r57\" (UID: \"f1f509b8-7e29-445b-ad25-dd54a1c6dc07\") " pod="openshift-dns/node-resolver-87r57"
Jul 17 10:57:26.545380 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544818    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-host-cni-bin\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.545380 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544842    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fc69e11f-7c22-4067-8169-886d341a4b70-etc-selinux\") pod \"aws-ebs-csi-driver-node-p9h59\" (UID: \"fc69e11f-7c22-4067-8169-886d341a4b70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59"
Jul 17 10:57:26.545380 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544917    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-system-cni-dir\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.545380 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544941    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-multus-cni-dir\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.545380 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544966    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60837033-c957-4341-8f68-9e70bf72cce6-multus-daemon-config\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.545380 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.544995    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-host-kubelet\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.545380 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545039    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgd6s\" (UniqueName: \"kubernetes.io/projected/fc69e11f-7c22-4067-8169-886d341a4b70-kube-api-access-bgd6s\") pod \"aws-ebs-csi-driver-node-p9h59\" (UID: \"fc69e11f-7c22-4067-8169-886d341a4b70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59"
Jul 17 10:57:26.545380 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545071    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73402c16-36b4-442c-9009-4d41bd9003fc-konnectivity-ca\") pod \"konnectivity-agent-kz6gd\" (UID: \"73402c16-36b4-442c-9009-4d41bd9003fc\") " pod="kube-system/konnectivity-agent-kz6gd"
Jul 17 10:57:26.545380 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545065    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/84ae7bd0-6126-491f-9126-2d64506af889-dbus\") pod \"global-pull-secret-syncer-b7fx4\" (UID: \"84ae7bd0-6126-491f-9126-2d64506af889\") " pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:26.545380 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545123    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72f5631e-c70d-47b8-9405-d8b986fd12e7-host\") pod \"node-ca-qqqf9\" (UID: \"72f5631e-c70d-47b8-9405-d8b986fd12e7\") " pod="openshift-image-registry/node-ca-qqqf9"
Jul 17 10:57:26.545380 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545166    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b3498a1-e561-4239-8878-ad9595ad312a-cnibin\") pod \"multus-additional-cni-plugins-pcg7j\" (UID: \"4b3498a1-e561-4239-8878-ad9595ad312a\") " pod="openshift-multus/multus-additional-cni-plugins-pcg7j"
Jul 17 10:57:26.545380 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545178    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b3498a1-e561-4239-8878-ad9595ad312a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pcg7j\" (UID: \"4b3498a1-e561-4239-8878-ad9595ad312a\") " pod="openshift-multus/multus-additional-cni-plugins-pcg7j"
Jul 17 10:57:26.545380 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545193    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-cnibin\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.545380 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545220    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72f5631e-c70d-47b8-9405-d8b986fd12e7-host\") pod \"node-ca-qqqf9\" (UID: \"72f5631e-c70d-47b8-9405-d8b986fd12e7\") " pod="openshift-image-registry/node-ca-qqqf9"
Jul 17 10:57:26.545380 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545223    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-host-run-k8s-cni-cncf-io\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.546260 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545229    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b3498a1-e561-4239-8878-ad9595ad312a-cnibin\") pod \"multus-additional-cni-plugins-pcg7j\" (UID: \"4b3498a1-e561-4239-8878-ad9595ad312a\") " pod="openshift-multus/multus-additional-cni-plugins-pcg7j"
Jul 17 10:57:26.546260 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545264    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-etc-openvswitch\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.546260 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545300    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-log-socket\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.546260 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545319    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fc69e11f-7c22-4067-8169-886d341a4b70-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p9h59\" (UID: \"fc69e11f-7c22-4067-8169-886d341a4b70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59"
Jul 17 10:57:26.546260 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545335    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-etc-sysctl-d\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.546260 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545356    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1f509b8-7e29-445b-ad25-dd54a1c6dc07-tmp-dir\") pod \"node-resolver-87r57\" (UID: \"f1f509b8-7e29-445b-ad25-dd54a1c6dc07\") " pod="openshift-dns/node-resolver-87r57"
Jul 17 10:57:26.546260 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545370    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-multus-conf-dir\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.546260 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545386    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fc69e11f-7c22-4067-8169-886d341a4b70-sys-fs\") pod \"aws-ebs-csi-driver-node-p9h59\" (UID: \"fc69e11f-7c22-4067-8169-886d341a4b70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59"
Jul 17 10:57:26.546260 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545406    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b3498a1-e561-4239-8878-ad9595ad312a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pcg7j\" (UID: \"4b3498a1-e561-4239-8878-ad9595ad312a\") " pod="openshift-multus/multus-additional-cni-plugins-pcg7j"
Jul 17 10:57:26.546260 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545442    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73402c16-36b4-442c-9009-4d41bd9003fc-agent-certs\") pod \"konnectivity-agent-kz6gd\" (UID: \"73402c16-36b4-442c-9009-4d41bd9003fc\") " pod="kube-system/konnectivity-agent-kz6gd"
Jul 17 10:57:26.546260 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545517    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-host-run-netns\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.546260 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545580    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b3498a1-e561-4239-8878-ad9595ad312a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pcg7j\" (UID: \"4b3498a1-e561-4239-8878-ad9595ad312a\") " pod="openshift-multus/multus-additional-cni-plugins-pcg7j"
Jul 17 10:57:26.546260 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545633    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/84ae7bd0-6126-491f-9126-2d64506af889-kubelet-config\") pod \"global-pull-secret-syncer-b7fx4\" (UID: \"84ae7bd0-6126-491f-9126-2d64506af889\") " pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:26.546260 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545706    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b3498a1-e561-4239-8878-ad9595ad312a-cni-binary-copy\") pod \"multus-additional-cni-plugins-pcg7j\" (UID: \"4b3498a1-e561-4239-8878-ad9595ad312a\") " pod="openshift-multus/multus-additional-cni-plugins-pcg7j"
Jul 17 10:57:26.546260 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d2304352-6b6e-42d3-af75-146ba8b85e0b-iptables-alerter-script\") pod \"iptables-alerter-pvvr8\" (UID: \"d2304352-6b6e-42d3-af75-146ba8b85e0b\") " pod="openshift-network-operator/iptables-alerter-pvvr8"
Jul 17 10:57:26.546260 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545786    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 17 10:57:26.546260 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545811    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-var-lib-openvswitch\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.547307 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545843    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-node-log\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.547307 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545871    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/55d1de37-8635-4de8-ab59-4ebfd0346a40-ovnkube-script-lib\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.547307 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545899    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84ae7bd0-6126-491f-9126-2d64506af889-original-pull-secret\") pod \"global-pull-secret-syncer-b7fx4\" (UID: \"84ae7bd0-6126-491f-9126-2d64506af889\") " pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:26.547307 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545945    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-etc-sysconfig\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.547307 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.545983    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-host-run-multus-certs\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.547307 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.546021    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.547307 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.546064    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/55d1de37-8635-4de8-ab59-4ebfd0346a40-env-overrides\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.547307 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.546101    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-sys\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.547307 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.546123    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w9ts\" (UniqueName: \"kubernetes.io/projected/609409ce-006b-440e-8403-dda1bef2dd5b-kube-api-access-2w9ts\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.547307 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.546146    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-host-run-netns\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.547307 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.546186    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-host-var-lib-cni-bin\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.547307 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.546229    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-host-var-lib-cni-multus\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.547307 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.546255    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h77c\" (UniqueName: \"kubernetes.io/projected/60837033-c957-4341-8f68-9e70bf72cce6-kube-api-access-5h77c\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.547307 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.546283    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/55d1de37-8635-4de8-ab59-4ebfd0346a40-ovnkube-config\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.547307 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.546347    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fc69e11f-7c22-4067-8169-886d341a4b70-device-dir\") pod \"aws-ebs-csi-driver-node-p9h59\" (UID: \"fc69e11f-7c22-4067-8169-886d341a4b70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59"
Jul 17 10:57:26.547307 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.546379    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-etc-sysctl-conf\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.547865 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.546406    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-var-lib-kubelet\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.547865 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.546431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-multus-socket-dir-parent\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.547865 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.546456    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60837033-c957-4341-8f68-9e70bf72cce6-cni-binary-copy\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.547865 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.546535    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2304352-6b6e-42d3-af75-146ba8b85e0b-host-slash\") pod \"iptables-alerter-pvvr8\" (UID: \"d2304352-6b6e-42d3-af75-146ba8b85e0b\") " pod="openshift-network-operator/iptables-alerter-pvvr8"
Jul 17 10:57:26.547865 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.546566    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4b3498a1-e561-4239-8878-ad9595ad312a-os-release\") pod \"multus-additional-cni-plugins-pcg7j\" (UID: \"4b3498a1-e561-4239-8878-ad9595ad312a\") " pod="openshift-multus/multus-additional-cni-plugins-pcg7j"
Jul 17 10:57:26.547865 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.546606    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9c3cb80-365e-4c10-bde0-1e9f17948a97-metrics-certs\") pod \"network-metrics-daemon-s5c9b\" (UID: \"e9c3cb80-365e-4c10-bde0-1e9f17948a97\") " pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:26.547865 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.546679    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4b3498a1-e561-4239-8878-ad9595ad312a-os-release\") pod \"multus-additional-cni-plugins-pcg7j\" (UID: \"4b3498a1-e561-4239-8878-ad9595ad312a\") " pod="openshift-multus/multus-additional-cni-plugins-pcg7j"
Jul 17 10:57:26.547865 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.546681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-systemd-units\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.547865 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:26.546726    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 10:57:26.547865 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.546764    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-run-openvswitch\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.547865 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.546794    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-host-cni-netd\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.547865 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.546821    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lgbt\" (UniqueName: \"kubernetes.io/projected/72f5631e-c70d-47b8-9405-d8b986fd12e7-kube-api-access-8lgbt\") pod \"node-ca-qqqf9\" (UID: \"72f5631e-c70d-47b8-9405-d8b986fd12e7\") " pod="openshift-image-registry/node-ca-qqqf9"
Jul 17 10:57:26.547865 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.546820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b3498a1-e561-4239-8878-ad9595ad312a-cni-binary-copy\") pod \"multus-additional-cni-plugins-pcg7j\" (UID: \"4b3498a1-e561-4239-8878-ad9595ad312a\") " pod="openshift-multus/multus-additional-cni-plugins-pcg7j"
Jul 17 10:57:26.547865 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:26.546861    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9c3cb80-365e-4c10-bde0-1e9f17948a97-metrics-certs podName:e9c3cb80-365e-4c10-bde0-1e9f17948a97 nodeName:}" failed. No retries permitted until 2026-07-17 10:57:27.046840095 +0000 UTC m=+1.976379717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9c3cb80-365e-4c10-bde0-1e9f17948a97-metrics-certs") pod "network-metrics-daemon-s5c9b" (UID: "e9c3cb80-365e-4c10-bde0-1e9f17948a97") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 10:57:26.547865 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.546954    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b3498a1-e561-4239-8878-ad9595ad312a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pcg7j\" (UID: \"4b3498a1-e561-4239-8878-ad9595ad312a\") " pod="openshift-multus/multus-additional-cni-plugins-pcg7j"
Jul 17 10:57:26.547865 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.547001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-etc-modprobe-d\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.548552 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.547051    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-os-release\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.548552 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.547137    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-host-run-ovn-kubernetes\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.548552 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.547167    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fc69e11f-7c22-4067-8169-886d341a4b70-registration-dir\") pod \"aws-ebs-csi-driver-node-p9h59\" (UID: \"fc69e11f-7c22-4067-8169-886d341a4b70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59"
Jul 17 10:57:26.548552 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.547488    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b3498a1-e561-4239-8878-ad9595ad312a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pcg7j\" (UID: \"4b3498a1-e561-4239-8878-ad9595ad312a\") " pod="openshift-multus/multus-additional-cni-plugins-pcg7j"
Jul 17 10:57:26.549459 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.549430    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73402c16-36b4-442c-9009-4d41bd9003fc-agent-certs\") pod \"konnectivity-agent-kz6gd\" (UID: \"73402c16-36b4-442c-9009-4d41bd9003fc\") " pod="kube-system/konnectivity-agent-kz6gd"
Jul 17 10:57:26.554417 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.554387    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxbqs\" (UniqueName: \"kubernetes.io/projected/e9c3cb80-365e-4c10-bde0-1e9f17948a97-kube-api-access-cxbqs\") pod \"network-metrics-daemon-s5c9b\" (UID: \"e9c3cb80-365e-4c10-bde0-1e9f17948a97\") " pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:26.554515 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.554414    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swx4c\" (UniqueName: \"kubernetes.io/projected/4b3498a1-e561-4239-8878-ad9595ad312a-kube-api-access-swx4c\") pod \"multus-additional-cni-plugins-pcg7j\" (UID: \"4b3498a1-e561-4239-8878-ad9595ad312a\") " pod="openshift-multus/multus-additional-cni-plugins-pcg7j"
Jul 17 10:57:26.556359 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.556329    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lgbt\" (UniqueName: \"kubernetes.io/projected/72f5631e-c70d-47b8-9405-d8b986fd12e7-kube-api-access-8lgbt\") pod \"node-ca-qqqf9\" (UID: \"72f5631e-c70d-47b8-9405-d8b986fd12e7\") " pod="openshift-image-registry/node-ca-qqqf9"
Jul 17 10:57:26.633475 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.633441    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 10:57:26.647604 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.647574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-host-cni-bin\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.647604 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.647607    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fc69e11f-7c22-4067-8169-886d341a4b70-etc-selinux\") pod \"aws-ebs-csi-driver-node-p9h59\" (UID: \"fc69e11f-7c22-4067-8169-886d341a4b70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59"
Jul 17 10:57:26.647825 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.647623    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-system-cni-dir\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.647825 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.647639    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-multus-cni-dir\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.647825 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.647656    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60837033-c957-4341-8f68-9e70bf72cce6-multus-daemon-config\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.647825 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.647674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-host-kubelet\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.647825 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.647697    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-host-cni-bin\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.647825 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.647697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgd6s\" (UniqueName: \"kubernetes.io/projected/fc69e11f-7c22-4067-8169-886d341a4b70-kube-api-access-bgd6s\") pod \"aws-ebs-csi-driver-node-p9h59\" (UID: \"fc69e11f-7c22-4067-8169-886d341a4b70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59"
Jul 17 10:57:26.647825 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.647746    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/84ae7bd0-6126-491f-9126-2d64506af889-dbus\") pod \"global-pull-secret-syncer-b7fx4\" (UID: \"84ae7bd0-6126-491f-9126-2d64506af889\") " pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:26.647825 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.647750    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fc69e11f-7c22-4067-8169-886d341a4b70-etc-selinux\") pod \"aws-ebs-csi-driver-node-p9h59\" (UID: \"fc69e11f-7c22-4067-8169-886d341a4b70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59"
Jul 17 10:57:26.647825 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.647749    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-host-kubelet\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.647825 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.647750    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-system-cni-dir\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.647825 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.647775    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-cnibin\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.647825 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.647820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-host-run-k8s-cni-cncf-io\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.648466 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.647849    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-etc-openvswitch\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.648466 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.647872    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-log-socket\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.648466 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.647869    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-cnibin\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.648466 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.647902    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-multus-cni-dir\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.648466 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.647918    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fc69e11f-7c22-4067-8169-886d341a4b70-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p9h59\" (UID: \"fc69e11f-7c22-4067-8169-886d341a4b70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59"
Jul 17 10:57:26.648466 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.647925    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-etc-openvswitch\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.648466 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.647941    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-etc-sysctl-d\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.648466 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.647944    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-log-socket\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.648466 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.647919    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-host-run-k8s-cni-cncf-io\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.648466 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.647871    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/84ae7bd0-6126-491f-9126-2d64506af889-dbus\") pod \"global-pull-secret-syncer-b7fx4\" (UID: \"84ae7bd0-6126-491f-9126-2d64506af889\") " pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:26.648466 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.647968    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1f509b8-7e29-445b-ad25-dd54a1c6dc07-tmp-dir\") pod \"node-resolver-87r57\" (UID: \"f1f509b8-7e29-445b-ad25-dd54a1c6dc07\") " pod="openshift-dns/node-resolver-87r57"
Jul 17 10:57:26.648466 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.647987    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fc69e11f-7c22-4067-8169-886d341a4b70-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p9h59\" (UID: \"fc69e11f-7c22-4067-8169-886d341a4b70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59"
Jul 17 10:57:26.648466 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.647991    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-multus-conf-dir\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.648466 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648035    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-multus-conf-dir\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.648466 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fc69e11f-7c22-4067-8169-886d341a4b70-sys-fs\") pod \"aws-ebs-csi-driver-node-p9h59\" (UID: \"fc69e11f-7c22-4067-8169-886d341a4b70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59"
Jul 17 10:57:26.648466 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648051    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-etc-sysctl-d\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.648466 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648095    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-host-run-netns\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.648466 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648119    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/84ae7bd0-6126-491f-9126-2d64506af889-kubelet-config\") pod \"global-pull-secret-syncer-b7fx4\" (UID: \"84ae7bd0-6126-491f-9126-2d64506af889\") " pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:26.649438 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648133    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-host-run-netns\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.649438 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648147    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fc69e11f-7c22-4067-8169-886d341a4b70-sys-fs\") pod \"aws-ebs-csi-driver-node-p9h59\" (UID: \"fc69e11f-7c22-4067-8169-886d341a4b70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59"
Jul 17 10:57:26.649438 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648140    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d2304352-6b6e-42d3-af75-146ba8b85e0b-iptables-alerter-script\") pod \"iptables-alerter-pvvr8\" (UID: \"d2304352-6b6e-42d3-af75-146ba8b85e0b\") " pod="openshift-network-operator/iptables-alerter-pvvr8"
Jul 17 10:57:26.649438 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648181    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/84ae7bd0-6126-491f-9126-2d64506af889-kubelet-config\") pod \"global-pull-secret-syncer-b7fx4\" (UID: \"84ae7bd0-6126-491f-9126-2d64506af889\") " pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:26.649438 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648192    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-var-lib-openvswitch\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.649438 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648227    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-node-log\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.649438 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648244    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/55d1de37-8635-4de8-ab59-4ebfd0346a40-ovnkube-script-lib\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.649438 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648262    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84ae7bd0-6126-491f-9126-2d64506af889-original-pull-secret\") pod \"global-pull-secret-syncer-b7fx4\" (UID: \"84ae7bd0-6126-491f-9126-2d64506af889\") " pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:26.649438 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648295    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-node-log\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.649438 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648305    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-etc-sysconfig\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.649438 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648321    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1f509b8-7e29-445b-ad25-dd54a1c6dc07-tmp-dir\") pod \"node-resolver-87r57\" (UID: \"f1f509b8-7e29-445b-ad25-dd54a1c6dc07\") " pod="openshift-dns/node-resolver-87r57"
Jul 17 10:57:26.649438 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648331    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-host-run-multus-certs\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.649438 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648336    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-var-lib-openvswitch\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.649438 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648361    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.649438 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648364    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60837033-c957-4341-8f68-9e70bf72cce6-multus-daemon-config\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.649438 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648387    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/55d1de37-8635-4de8-ab59-4ebfd0346a40-env-overrides\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.649438 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648413    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-sys\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.650318 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648415    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-host-run-multus-certs\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.650318 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:26.648421    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 10:57:26.650318 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648434    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-etc-sysconfig\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.650318 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648437    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w9ts\" (UniqueName: \"kubernetes.io/projected/609409ce-006b-440e-8403-dda1bef2dd5b-kube-api-access-2w9ts\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.650318 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648454    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.650318 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648482    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-host-run-netns\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.650318 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:26.648499    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84ae7bd0-6126-491f-9126-2d64506af889-original-pull-secret podName:84ae7bd0-6126-491f-9126-2d64506af889 nodeName:}" failed. No retries permitted until 2026-07-17 10:57:27.148478952 +0000 UTC m=+2.078018581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84ae7bd0-6126-491f-9126-2d64506af889-original-pull-secret") pod "global-pull-secret-syncer-b7fx4" (UID: "84ae7bd0-6126-491f-9126-2d64506af889") : object "kube-system"/"original-pull-secret" not registered
Jul 17 10:57:26.650318 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648518    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-host-run-netns\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.650318 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648571    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-sys\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.650318 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648612    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-host-var-lib-cni-bin\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.650318 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648641    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-host-var-lib-cni-multus\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.650318 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648668    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h77c\" (UniqueName: \"kubernetes.io/projected/60837033-c957-4341-8f68-9e70bf72cce6-kube-api-access-5h77c\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.650318 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648671    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-host-var-lib-cni-bin\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.650318 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648692    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/55d1de37-8635-4de8-ab59-4ebfd0346a40-ovnkube-config\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.650318 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648731    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fc69e11f-7c22-4067-8169-886d341a4b70-device-dir\") pod \"aws-ebs-csi-driver-node-p9h59\" (UID: \"fc69e11f-7c22-4067-8169-886d341a4b70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59"
Jul 17 10:57:26.650318 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648760    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-etc-sysctl-conf\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.650318 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648766    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-host-var-lib-cni-multus\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.651247 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648784    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-var-lib-kubelet\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.651247 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648810    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-multus-socket-dir-parent\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.651247 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648824    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fc69e11f-7c22-4067-8169-886d341a4b70-device-dir\") pod \"aws-ebs-csi-driver-node-p9h59\" (UID: \"fc69e11f-7c22-4067-8169-886d341a4b70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59"
Jul 17 10:57:26.651247 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648839    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60837033-c957-4341-8f68-9e70bf72cce6-cni-binary-copy\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.651247 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648863    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2304352-6b6e-42d3-af75-146ba8b85e0b-host-slash\") pod \"iptables-alerter-pvvr8\" (UID: \"d2304352-6b6e-42d3-af75-146ba8b85e0b\") " pod="openshift-network-operator/iptables-alerter-pvvr8"
Jul 17 10:57:26.651247 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648873    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-var-lib-kubelet\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.651247 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648883    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/55d1de37-8635-4de8-ab59-4ebfd0346a40-ovnkube-script-lib\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.651247 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648890    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/55d1de37-8635-4de8-ab59-4ebfd0346a40-env-overrides\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.651247 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-systemd-units\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.651247 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648929    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-run-openvswitch\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.651247 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648950    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-multus-socket-dir-parent\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.651247 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648953    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-host-cni-netd\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.651247 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-etc-modprobe-d\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.651247 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648999    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-etc-sysctl-conf\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.651247 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649016    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-os-release\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.651247 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649023    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-run-openvswitch\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.651247 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648782    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d2304352-6b6e-42d3-af75-146ba8b85e0b-iptables-alerter-script\") pod \"iptables-alerter-pvvr8\" (UID: \"d2304352-6b6e-42d3-af75-146ba8b85e0b\") " pod="openshift-network-operator/iptables-alerter-pvvr8"
Jul 17 10:57:26.652073 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649039    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-systemd-units\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.652073 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.648991    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-host-cni-netd\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.652073 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649055    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2304352-6b6e-42d3-af75-146ba8b85e0b-host-slash\") pod \"iptables-alerter-pvvr8\" (UID: \"d2304352-6b6e-42d3-af75-146ba8b85e0b\") " pod="openshift-network-operator/iptables-alerter-pvvr8"
Jul 17 10:57:26.652073 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649087    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-os-release\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.652073 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649090    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-host-run-ovn-kubernetes\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.652073 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649125    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-host-run-ovn-kubernetes\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.652073 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649144    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fc69e11f-7c22-4067-8169-886d341a4b70-registration-dir\") pod \"aws-ebs-csi-driver-node-p9h59\" (UID: \"fc69e11f-7c22-4067-8169-886d341a4b70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59"
Jul 17 10:57:26.652073 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649155    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-etc-modprobe-d\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.652073 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649175    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/55d1de37-8635-4de8-ab59-4ebfd0346a40-ovnkube-config\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.652073 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649188    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-etc-systemd\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.652073 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649221    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/609409ce-006b-440e-8403-dda1bef2dd5b-tmp\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.652073 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649219    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-etc-systemd\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.652073 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649187    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fc69e11f-7c22-4067-8169-886d341a4b70-registration-dir\") pod \"aws-ebs-csi-driver-node-p9h59\" (UID: \"fc69e11f-7c22-4067-8169-886d341a4b70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59"
Jul 17 10:57:26.652073 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649247    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f1f509b8-7e29-445b-ad25-dd54a1c6dc07-hosts-file\") pod \"node-resolver-87r57\" (UID: \"f1f509b8-7e29-445b-ad25-dd54a1c6dc07\") " pod="openshift-dns/node-resolver-87r57"
Jul 17 10:57:26.652073 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649274    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-host-var-lib-kubelet\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.652073 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649301    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-run-systemd\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.652073 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649304    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f1f509b8-7e29-445b-ad25-dd54a1c6dc07-hosts-file\") pod \"node-resolver-87r57\" (UID: \"f1f509b8-7e29-445b-ad25-dd54a1c6dc07\") " pod="openshift-dns/node-resolver-87r57"
Jul 17 10:57:26.652073 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649327    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/55d1de37-8635-4de8-ab59-4ebfd0346a40-ovn-node-metrics-cert\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.652659 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649341    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-host-var-lib-kubelet\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.652659 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649354    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-etc-kubernetes\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.652659 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649390    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60837033-c957-4341-8f68-9e70bf72cce6-cni-binary-copy\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.652659 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649410    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-lib-modules\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.652659 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-run-systemd\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.652659 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649466    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-etc-kubernetes\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.652659 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649503    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-host\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.652659 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649508    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-lib-modules\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.652659 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/609409ce-006b-440e-8403-dda1bef2dd5b-etc-tuned\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.652659 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649555    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-etc-kubernetes\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.652659 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649579    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-run-ovn\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.652659 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649605    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fc69e11f-7c22-4067-8169-886d341a4b70-socket-dir\") pod \"aws-ebs-csi-driver-node-p9h59\" (UID: \"fc69e11f-7c22-4067-8169-886d341a4b70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59"
Jul 17 10:57:26.652659 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649636    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9v8m\" (UniqueName: \"kubernetes.io/projected/f2830f09-58f3-425c-a831-e5e34b542e78-kube-api-access-n9v8m\") pod \"network-check-target-g5hnv\" (UID: \"f2830f09-58f3-425c-a831-e5e34b542e78\") " pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:57:26.652659 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649665    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-run\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.652659 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649688    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-hostroot\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.652659 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649687    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-host\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.652659 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r76hn\" (UniqueName: \"kubernetes.io/projected/d2304352-6b6e-42d3-af75-146ba8b85e0b-kube-api-access-r76hn\") pod \"iptables-alerter-pvvr8\" (UID: \"d2304352-6b6e-42d3-af75-146ba8b85e0b\") " pod="openshift-network-operator/iptables-alerter-pvvr8"
Jul 17 10:57:26.652659 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649734    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fc69e11f-7c22-4067-8169-886d341a4b70-socket-dir\") pod \"aws-ebs-csi-driver-node-p9h59\" (UID: \"fc69e11f-7c22-4067-8169-886d341a4b70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59"
Jul 17 10:57:26.653239 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649741    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-run-ovn\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.653239 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649736    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-host-slash\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.653239 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649781    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55d1de37-8635-4de8-ab59-4ebfd0346a40-host-slash\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.653239 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649785    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfh6w\" (UniqueName: \"kubernetes.io/projected/55d1de37-8635-4de8-ab59-4ebfd0346a40-kube-api-access-cfh6w\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.653239 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649800    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/609409ce-006b-440e-8403-dda1bef2dd5b-run\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.653239 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649842    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcrfj\" (UniqueName: \"kubernetes.io/projected/f1f509b8-7e29-445b-ad25-dd54a1c6dc07-kube-api-access-mcrfj\") pod \"node-resolver-87r57\" (UID: \"f1f509b8-7e29-445b-ad25-dd54a1c6dc07\") " pod="openshift-dns/node-resolver-87r57"
Jul 17 10:57:26.653239 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.649843    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-hostroot\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.653239 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.650066    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60837033-c957-4341-8f68-9e70bf72cce6-etc-kubernetes\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.653239 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.651818    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/55d1de37-8635-4de8-ab59-4ebfd0346a40-ovn-node-metrics-cert\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.653239 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.651965    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/609409ce-006b-440e-8403-dda1bef2dd5b-tmp\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.653239 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.652054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/609409ce-006b-440e-8403-dda1bef2dd5b-etc-tuned\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.660572 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:26.660546    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 10:57:26.660715 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:26.660580    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 10:57:26.660715 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:26.660595    2563 projected.go:196] Error preparing data for projected volume kube-api-access-n9v8m for pod openshift-network-diagnostics/network-check-target-g5hnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 10:57:26.660715 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:26.660671    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2830f09-58f3-425c-a831-e5e34b542e78-kube-api-access-n9v8m podName:f2830f09-58f3-425c-a831-e5e34b542e78 nodeName:}" failed. No retries permitted until 2026-07-17 10:57:27.160650135 +0000 UTC m=+2.090189779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-n9v8m" (UniqueName: "kubernetes.io/projected/f2830f09-58f3-425c-a831-e5e34b542e78-kube-api-access-n9v8m") pod "network-check-target-g5hnv" (UID: "f2830f09-58f3-425c-a831-e5e34b542e78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 10:57:26.660944 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.660831    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h77c\" (UniqueName: \"kubernetes.io/projected/60837033-c957-4341-8f68-9e70bf72cce6-kube-api-access-5h77c\") pod \"multus-j4mp9\" (UID: \"60837033-c957-4341-8f68-9e70bf72cce6\") " pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.661133 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.661111    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgd6s\" (UniqueName: \"kubernetes.io/projected/fc69e11f-7c22-4067-8169-886d341a4b70-kube-api-access-bgd6s\") pod \"aws-ebs-csi-driver-node-p9h59\" (UID: \"fc69e11f-7c22-4067-8169-886d341a4b70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59"
Jul 17 10:57:26.662783 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.662762    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w9ts\" (UniqueName: \"kubernetes.io/projected/609409ce-006b-440e-8403-dda1bef2dd5b-kube-api-access-2w9ts\") pod \"tuned-l426v\" (UID: \"609409ce-006b-440e-8403-dda1bef2dd5b\") " pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.663405 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.663386    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcrfj\" (UniqueName: \"kubernetes.io/projected/f1f509b8-7e29-445b-ad25-dd54a1c6dc07-kube-api-access-mcrfj\") pod \"node-resolver-87r57\" (UID: \"f1f509b8-7e29-445b-ad25-dd54a1c6dc07\") " pod="openshift-dns/node-resolver-87r57"
Jul 17 10:57:26.663640 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.663615    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r76hn\" (UniqueName: \"kubernetes.io/projected/d2304352-6b6e-42d3-af75-146ba8b85e0b-kube-api-access-r76hn\") pod \"iptables-alerter-pvvr8\" (UID: \"d2304352-6b6e-42d3-af75-146ba8b85e0b\") " pod="openshift-network-operator/iptables-alerter-pvvr8"
Jul 17 10:57:26.663894 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.663878    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfh6w\" (UniqueName: \"kubernetes.io/projected/55d1de37-8635-4de8-ab59-4ebfd0346a40-kube-api-access-cfh6w\") pod \"ovnkube-node-nj5mj\" (UID: \"55d1de37-8635-4de8-ab59-4ebfd0346a40\") " pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.665692 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:26.665657    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098d9536b1eb81f9184c22a2bb857164.slice/crio-e552a5ec71fb34d5cd33f7395ebb301d175163fedf160f72d76b76209429c2de WatchSource:0}: Error finding container e552a5ec71fb34d5cd33f7395ebb301d175163fedf160f72d76b76209429c2de: Status 404 returned error can't find the container with id e552a5ec71fb34d5cd33f7395ebb301d175163fedf160f72d76b76209429c2de
Jul 17 10:57:26.671909 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.671891    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 10:57:26.685423 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:26.685397    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c42ae2a6854f18bdedf80ad478b60e.slice/crio-fe7c06472b0353191dc58ed828b75ae2aa541a9597584b2150dee693480c6867 WatchSource:0}: Error finding container fe7c06472b0353191dc58ed828b75ae2aa541a9597584b2150dee693480c6867: Status 404 returned error can't find the container with id fe7c06472b0353191dc58ed828b75ae2aa541a9597584b2150dee693480c6867
Jul 17 10:57:26.736344 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.736308    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 10:57:26.766842 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.766799    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qqqf9"
Jul 17 10:57:26.774817 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:26.774784    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f5631e_c70d_47b8_9405_d8b986fd12e7.slice/crio-ee9ada8e3da221910112f9e3873fa66857ffa31e86b9ada71649e61749e7da88 WatchSource:0}: Error finding container ee9ada8e3da221910112f9e3873fa66857ffa31e86b9ada71649e61749e7da88: Status 404 returned error can't find the container with id ee9ada8e3da221910112f9e3873fa66857ffa31e86b9ada71649e61749e7da88
Jul 17 10:57:26.783212 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.783188    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pcg7j"
Jul 17 10:57:26.791516 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:26.791488    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b3498a1_e561_4239_8878_ad9595ad312a.slice/crio-c19a99f32176e2193a14c325c1045b4899dad9d43385a3841a52a8440a10a94b WatchSource:0}: Error finding container c19a99f32176e2193a14c325c1045b4899dad9d43385a3841a52a8440a10a94b: Status 404 returned error can't find the container with id c19a99f32176e2193a14c325c1045b4899dad9d43385a3841a52a8440a10a94b
Jul 17 10:57:26.798914 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.798877    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kz6gd"
Jul 17 10:57:26.807097 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:26.807056    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73402c16_36b4_442c_9009_4d41bd9003fc.slice/crio-11e8714c3f7111380d0000acc3db260beef48a7046d2bd9a933f275d6f08f2ac WatchSource:0}: Error finding container 11e8714c3f7111380d0000acc3db260beef48a7046d2bd9a933f275d6f08f2ac: Status 404 returned error can't find the container with id 11e8714c3f7111380d0000acc3db260beef48a7046d2bd9a933f275d6f08f2ac
Jul 17 10:57:26.819204 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.819182    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j4mp9"
Jul 17 10:57:26.826894 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.826874    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pvvr8"
Jul 17 10:57:26.827391 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:26.827368    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60837033_c957_4341_8f68_9e70bf72cce6.slice/crio-93f3a6a1fbaab9ae3206d142f456a326f6d23653ac72cf247ebf236cd45e86c3 WatchSource:0}: Error finding container 93f3a6a1fbaab9ae3206d142f456a326f6d23653ac72cf247ebf236cd45e86c3: Status 404 returned error can't find the container with id 93f3a6a1fbaab9ae3206d142f456a326f6d23653ac72cf247ebf236cd45e86c3
Jul 17 10:57:26.834519 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:26.834494    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2304352_6b6e_42d3_af75_146ba8b85e0b.slice/crio-3878f8f955de22d136a13e6d9eda8a1485349dad14c58c617f870cbf7be8e474 WatchSource:0}: Error finding container 3878f8f955de22d136a13e6d9eda8a1485349dad14c58c617f870cbf7be8e474: Status 404 returned error can't find the container with id 3878f8f955de22d136a13e6d9eda8a1485349dad14c58c617f870cbf7be8e474
Jul 17 10:57:26.844531 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.844509    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:26.853084 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:26.853061    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d1de37_8635_4de8_ab59_4ebfd0346a40.slice/crio-26ff53477d8131f62e5d2069dcf6b2928cd91df7896d00b79f61b06ed871696f WatchSource:0}: Error finding container 26ff53477d8131f62e5d2069dcf6b2928cd91df7896d00b79f61b06ed871696f: Status 404 returned error can't find the container with id 26ff53477d8131f62e5d2069dcf6b2928cd91df7896d00b79f61b06ed871696f
Jul 17 10:57:26.864292 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.864272    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59"
Jul 17 10:57:26.872229 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:26.872204    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc69e11f_7c22_4067_8169_886d341a4b70.slice/crio-bcab55b30c10a69f5691f3e9811ce721e77b7793bfa71b521c437da38ddbb2bc WatchSource:0}: Error finding container bcab55b30c10a69f5691f3e9811ce721e77b7793bfa71b521c437da38ddbb2bc: Status 404 returned error can't find the container with id bcab55b30c10a69f5691f3e9811ce721e77b7793bfa71b521c437da38ddbb2bc
Jul 17 10:57:26.882051 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.882029    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l426v"
Jul 17 10:57:26.888524 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:26.888503    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-87r57"
Jul 17 10:57:26.890194 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:26.890168    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod609409ce_006b_440e_8403_dda1bef2dd5b.slice/crio-3a9e3c3567d7f6be03280836a6b6a63c1564a43b2b87bdaee9d388d9c2b9de69 WatchSource:0}: Error finding container 3a9e3c3567d7f6be03280836a6b6a63c1564a43b2b87bdaee9d388d9c2b9de69: Status 404 returned error can't find the container with id 3a9e3c3567d7f6be03280836a6b6a63c1564a43b2b87bdaee9d388d9c2b9de69
Jul 17 10:57:26.898059 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:57:26.898030    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f509b8_7e29_445b_ad25_dd54a1c6dc07.slice/crio-7b76e8e52072827c22f00e18d48e39281f68c9491c351d2bfd3f56fcbf3520d8 WatchSource:0}: Error finding container 7b76e8e52072827c22f00e18d48e39281f68c9491c351d2bfd3f56fcbf3520d8: Status 404 returned error can't find the container with id 7b76e8e52072827c22f00e18d48e39281f68c9491c351d2bfd3f56fcbf3520d8
Jul 17 10:57:27.053687 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:27.053646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9c3cb80-365e-4c10-bde0-1e9f17948a97-metrics-certs\") pod \"network-metrics-daemon-s5c9b\" (UID: \"e9c3cb80-365e-4c10-bde0-1e9f17948a97\") " pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:27.053836 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:27.053781    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 10:57:27.053872 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:27.053838    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9c3cb80-365e-4c10-bde0-1e9f17948a97-metrics-certs podName:e9c3cb80-365e-4c10-bde0-1e9f17948a97 nodeName:}" failed. No retries permitted until 2026-07-17 10:57:28.053820978 +0000 UTC m=+2.983360615 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9c3cb80-365e-4c10-bde0-1e9f17948a97-metrics-certs") pod "network-metrics-daemon-s5c9b" (UID: "e9c3cb80-365e-4c10-bde0-1e9f17948a97") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 10:57:27.154554 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:27.154462    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84ae7bd0-6126-491f-9126-2d64506af889-original-pull-secret\") pod \"global-pull-secret-syncer-b7fx4\" (UID: \"84ae7bd0-6126-491f-9126-2d64506af889\") " pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:27.154707 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:27.154592    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 10:57:27.154707 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:27.154646    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84ae7bd0-6126-491f-9126-2d64506af889-original-pull-secret podName:84ae7bd0-6126-491f-9126-2d64506af889 nodeName:}" failed. No retries permitted until 2026-07-17 10:57:28.154633722 +0000 UTC m=+3.084173342 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84ae7bd0-6126-491f-9126-2d64506af889-original-pull-secret") pod "global-pull-secret-syncer-b7fx4" (UID: "84ae7bd0-6126-491f-9126-2d64506af889") : object "kube-system"/"original-pull-secret" not registered
Jul 17 10:57:27.174001 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:27.173952    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 10:57:27.255440 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:27.255405    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9v8m\" (UniqueName: \"kubernetes.io/projected/f2830f09-58f3-425c-a831-e5e34b542e78-kube-api-access-n9v8m\") pod \"network-check-target-g5hnv\" (UID: \"f2830f09-58f3-425c-a831-e5e34b542e78\") " pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:57:27.255632 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:27.255588    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 10:57:27.255632 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:27.255613    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 10:57:27.255632 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:27.255624    2563 projected.go:196] Error preparing data for projected volume kube-api-access-n9v8m for pod openshift-network-diagnostics/network-check-target-g5hnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 10:57:27.255810 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:27.255684    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2830f09-58f3-425c-a831-e5e34b542e78-kube-api-access-n9v8m podName:f2830f09-58f3-425c-a831-e5e34b542e78 nodeName:}" failed. No retries permitted until 2026-07-17 10:57:28.255667629 +0000 UTC m=+3.185207250 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-n9v8m" (UniqueName: "kubernetes.io/projected/f2830f09-58f3-425c-a831-e5e34b542e78-kube-api-access-n9v8m") pod "network-check-target-g5hnv" (UID: "f2830f09-58f3-425c-a831-e5e34b542e78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 10:57:27.517200 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:27.516997    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 10:52:26 +0000 UTC" deadline="2028-01-08 22:45:11.443280738 +0000 UTC"
Jul 17 10:57:27.517200 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:27.517066    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12971h47m43.926219698s"
Jul 17 10:57:27.662601 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:27.662569    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:27.662784 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:27.662706    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c9b" podUID="e9c3cb80-365e-4c10-bde0-1e9f17948a97"
Jul 17 10:57:27.683909 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:27.683842    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcg7j" event={"ID":"4b3498a1-e561-4239-8878-ad9595ad312a","Type":"ContainerStarted","Data":"c19a99f32176e2193a14c325c1045b4899dad9d43385a3841a52a8440a10a94b"}
Jul 17 10:57:27.701102 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:27.700992    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qqqf9" event={"ID":"72f5631e-c70d-47b8-9405-d8b986fd12e7","Type":"ContainerStarted","Data":"ee9ada8e3da221910112f9e3873fa66857ffa31e86b9ada71649e61749e7da88"}
Jul 17 10:57:27.709024 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:27.708988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-233.ec2.internal" event={"ID":"d2c42ae2a6854f18bdedf80ad478b60e","Type":"ContainerStarted","Data":"fe7c06472b0353191dc58ed828b75ae2aa541a9597584b2150dee693480c6867"}
Jul 17 10:57:27.715350 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:27.715306    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj" event={"ID":"55d1de37-8635-4de8-ab59-4ebfd0346a40","Type":"ContainerStarted","Data":"26ff53477d8131f62e5d2069dcf6b2928cd91df7896d00b79f61b06ed871696f"}
Jul 17 10:57:27.736225 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:27.736092    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pvvr8" event={"ID":"d2304352-6b6e-42d3-af75-146ba8b85e0b","Type":"ContainerStarted","Data":"3878f8f955de22d136a13e6d9eda8a1485349dad14c58c617f870cbf7be8e474"}
Jul 17 10:57:27.748767 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:27.748728    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-233.ec2.internal" event={"ID":"098d9536b1eb81f9184c22a2bb857164","Type":"ContainerStarted","Data":"e552a5ec71fb34d5cd33f7395ebb301d175163fedf160f72d76b76209429c2de"}
Jul 17 10:57:27.765161 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:27.761168    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-87r57" event={"ID":"f1f509b8-7e29-445b-ad25-dd54a1c6dc07","Type":"ContainerStarted","Data":"7b76e8e52072827c22f00e18d48e39281f68c9491c351d2bfd3f56fcbf3520d8"}
Jul 17 10:57:27.771838 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:27.771752    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l426v" event={"ID":"609409ce-006b-440e-8403-dda1bef2dd5b","Type":"ContainerStarted","Data":"3a9e3c3567d7f6be03280836a6b6a63c1564a43b2b87bdaee9d388d9c2b9de69"}
Jul 17 10:57:27.784743 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:27.784704    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59" event={"ID":"fc69e11f-7c22-4067-8169-886d341a4b70","Type":"ContainerStarted","Data":"bcab55b30c10a69f5691f3e9811ce721e77b7793bfa71b521c437da38ddbb2bc"}
Jul 17 10:57:27.798656 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:27.798614    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j4mp9" event={"ID":"60837033-c957-4341-8f68-9e70bf72cce6","Type":"ContainerStarted","Data":"93f3a6a1fbaab9ae3206d142f456a326f6d23653ac72cf247ebf236cd45e86c3"}
Jul 17 10:57:27.813469 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:27.813396    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kz6gd" event={"ID":"73402c16-36b4-442c-9009-4d41bd9003fc","Type":"ContainerStarted","Data":"11e8714c3f7111380d0000acc3db260beef48a7046d2bd9a933f275d6f08f2ac"}
Jul 17 10:57:28.063970 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:28.063851    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9c3cb80-365e-4c10-bde0-1e9f17948a97-metrics-certs\") pod \"network-metrics-daemon-s5c9b\" (UID: \"e9c3cb80-365e-4c10-bde0-1e9f17948a97\") " pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:28.064175 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:28.064060    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 10:57:28.064175 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:28.064141    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9c3cb80-365e-4c10-bde0-1e9f17948a97-metrics-certs podName:e9c3cb80-365e-4c10-bde0-1e9f17948a97 nodeName:}" failed. No retries permitted until 2026-07-17 10:57:30.064107646 +0000 UTC m=+4.993647284 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9c3cb80-365e-4c10-bde0-1e9f17948a97-metrics-certs") pod "network-metrics-daemon-s5c9b" (UID: "e9c3cb80-365e-4c10-bde0-1e9f17948a97") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 10:57:28.165021 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:28.164977    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84ae7bd0-6126-491f-9126-2d64506af889-original-pull-secret\") pod \"global-pull-secret-syncer-b7fx4\" (UID: \"84ae7bd0-6126-491f-9126-2d64506af889\") " pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:28.165240 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:28.165174    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 10:57:28.165240 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:28.165242    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84ae7bd0-6126-491f-9126-2d64506af889-original-pull-secret podName:84ae7bd0-6126-491f-9126-2d64506af889 nodeName:}" failed. No retries permitted until 2026-07-17 10:57:30.165222761 +0000 UTC m=+5.094762384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84ae7bd0-6126-491f-9126-2d64506af889-original-pull-secret") pod "global-pull-secret-syncer-b7fx4" (UID: "84ae7bd0-6126-491f-9126-2d64506af889") : object "kube-system"/"original-pull-secret" not registered
Jul 17 10:57:28.266246 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:28.265576    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9v8m\" (UniqueName: \"kubernetes.io/projected/f2830f09-58f3-425c-a831-e5e34b542e78-kube-api-access-n9v8m\") pod \"network-check-target-g5hnv\" (UID: \"f2830f09-58f3-425c-a831-e5e34b542e78\") " pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:57:28.266246 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:28.265752    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 10:57:28.266246 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:28.265775    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 10:57:28.266246 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:28.265789    2563 projected.go:196] Error preparing data for projected volume kube-api-access-n9v8m for pod openshift-network-diagnostics/network-check-target-g5hnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 10:57:28.266246 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:28.265850    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2830f09-58f3-425c-a831-e5e34b542e78-kube-api-access-n9v8m podName:f2830f09-58f3-425c-a831-e5e34b542e78 nodeName:}" failed. No retries permitted until 2026-07-17 10:57:30.265831545 +0000 UTC m=+5.195371168 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-n9v8m" (UniqueName: "kubernetes.io/projected/f2830f09-58f3-425c-a831-e5e34b542e78-kube-api-access-n9v8m") pod "network-check-target-g5hnv" (UID: "f2830f09-58f3-425c-a831-e5e34b542e78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 10:57:28.518145 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:28.518024    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 10:52:26 +0000 UTC" deadline="2027-12-27 17:47:55.244250077 +0000 UTC"
Jul 17 10:57:28.518145 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:28.518067    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12678h50m26.726188896s"
Jul 17 10:57:28.660050 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:28.660002    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:57:28.660260 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:28.660161    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5hnv" podUID="f2830f09-58f3-425c-a831-e5e34b542e78"
Jul 17 10:57:28.660675 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:28.660633    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:28.660789 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:28.660761    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7fx4" podUID="84ae7bd0-6126-491f-9126-2d64506af889"
Jul 17 10:57:28.808456 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:28.808175    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 10:57:29.662542 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:29.662499    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:29.663038 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:29.662655    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c9b" podUID="e9c3cb80-365e-4c10-bde0-1e9f17948a97"
Jul 17 10:57:30.080621 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:30.080574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9c3cb80-365e-4c10-bde0-1e9f17948a97-metrics-certs\") pod \"network-metrics-daemon-s5c9b\" (UID: \"e9c3cb80-365e-4c10-bde0-1e9f17948a97\") " pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:30.080831 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:30.080723    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 10:57:30.080831 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:30.080805    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9c3cb80-365e-4c10-bde0-1e9f17948a97-metrics-certs podName:e9c3cb80-365e-4c10-bde0-1e9f17948a97 nodeName:}" failed. No retries permitted until 2026-07-17 10:57:34.080786264 +0000 UTC m=+9.010325900 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9c3cb80-365e-4c10-bde0-1e9f17948a97-metrics-certs") pod "network-metrics-daemon-s5c9b" (UID: "e9c3cb80-365e-4c10-bde0-1e9f17948a97") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 10:57:30.181451 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:30.181407    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84ae7bd0-6126-491f-9126-2d64506af889-original-pull-secret\") pod \"global-pull-secret-syncer-b7fx4\" (UID: \"84ae7bd0-6126-491f-9126-2d64506af889\") " pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:30.181649 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:30.181623    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 10:57:30.181766 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:30.181721    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84ae7bd0-6126-491f-9126-2d64506af889-original-pull-secret podName:84ae7bd0-6126-491f-9126-2d64506af889 nodeName:}" failed. No retries permitted until 2026-07-17 10:57:34.181682761 +0000 UTC m=+9.111222385 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84ae7bd0-6126-491f-9126-2d64506af889-original-pull-secret") pod "global-pull-secret-syncer-b7fx4" (UID: "84ae7bd0-6126-491f-9126-2d64506af889") : object "kube-system"/"original-pull-secret" not registered
Jul 17 10:57:30.282962 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:30.282921    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9v8m\" (UniqueName: \"kubernetes.io/projected/f2830f09-58f3-425c-a831-e5e34b542e78-kube-api-access-n9v8m\") pod \"network-check-target-g5hnv\" (UID: \"f2830f09-58f3-425c-a831-e5e34b542e78\") " pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:57:30.283242 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:30.283222    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 10:57:30.283328 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:30.283249    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 10:57:30.283328 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:30.283263    2563 projected.go:196] Error preparing data for projected volume kube-api-access-n9v8m for pod openshift-network-diagnostics/network-check-target-g5hnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 10:57:30.283328 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:30.283325    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2830f09-58f3-425c-a831-e5e34b542e78-kube-api-access-n9v8m podName:f2830f09-58f3-425c-a831-e5e34b542e78 nodeName:}" failed. No retries permitted until 2026-07-17 10:57:34.283304506 +0000 UTC m=+9.212844130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-n9v8m" (UniqueName: "kubernetes.io/projected/f2830f09-58f3-425c-a831-e5e34b542e78-kube-api-access-n9v8m") pod "network-check-target-g5hnv" (UID: "f2830f09-58f3-425c-a831-e5e34b542e78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 10:57:30.659881 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:30.659827    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:57:30.660084 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:30.659960    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5hnv" podUID="f2830f09-58f3-425c-a831-e5e34b542e78"
Jul 17 10:57:30.660401 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:30.660379    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:30.660611 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:30.660496    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7fx4" podUID="84ae7bd0-6126-491f-9126-2d64506af889"
Jul 17 10:57:31.660780 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:31.660235    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:31.660780 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:31.660385    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c9b" podUID="e9c3cb80-365e-4c10-bde0-1e9f17948a97"
Jul 17 10:57:32.659591 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:32.659551    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:57:32.659786 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:32.659567    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:32.659786 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:32.659689    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5hnv" podUID="f2830f09-58f3-425c-a831-e5e34b542e78"
Jul 17 10:57:32.659786 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:32.659763    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7fx4" podUID="84ae7bd0-6126-491f-9126-2d64506af889"
Jul 17 10:57:33.660089 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:33.659947    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:33.660475 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:33.660121    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c9b" podUID="e9c3cb80-365e-4c10-bde0-1e9f17948a97"
Jul 17 10:57:34.114875 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:34.114786    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9c3cb80-365e-4c10-bde0-1e9f17948a97-metrics-certs\") pod \"network-metrics-daemon-s5c9b\" (UID: \"e9c3cb80-365e-4c10-bde0-1e9f17948a97\") " pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:34.115040 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:34.114976    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 10:57:34.115125 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:34.115055    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9c3cb80-365e-4c10-bde0-1e9f17948a97-metrics-certs podName:e9c3cb80-365e-4c10-bde0-1e9f17948a97 nodeName:}" failed. No retries permitted until 2026-07-17 10:57:42.115034627 +0000 UTC m=+17.044574255 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9c3cb80-365e-4c10-bde0-1e9f17948a97-metrics-certs") pod "network-metrics-daemon-s5c9b" (UID: "e9c3cb80-365e-4c10-bde0-1e9f17948a97") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 10:57:34.216158 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:34.216122    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84ae7bd0-6126-491f-9126-2d64506af889-original-pull-secret\") pod \"global-pull-secret-syncer-b7fx4\" (UID: \"84ae7bd0-6126-491f-9126-2d64506af889\") " pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:34.216359 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:34.216288    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 10:57:34.216359 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:34.216351    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84ae7bd0-6126-491f-9126-2d64506af889-original-pull-secret podName:84ae7bd0-6126-491f-9126-2d64506af889 nodeName:}" failed. No retries permitted until 2026-07-17 10:57:42.21633374 +0000 UTC m=+17.145873365 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84ae7bd0-6126-491f-9126-2d64506af889-original-pull-secret") pod "global-pull-secret-syncer-b7fx4" (UID: "84ae7bd0-6126-491f-9126-2d64506af889") : object "kube-system"/"original-pull-secret" not registered
Jul 17 10:57:34.318052 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:34.317323    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9v8m\" (UniqueName: \"kubernetes.io/projected/f2830f09-58f3-425c-a831-e5e34b542e78-kube-api-access-n9v8m\") pod \"network-check-target-g5hnv\" (UID: \"f2830f09-58f3-425c-a831-e5e34b542e78\") " pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:57:34.318052 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:34.317550    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 10:57:34.318052 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:34.317571    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 10:57:34.318052 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:34.317584    2563 projected.go:196] Error preparing data for projected volume kube-api-access-n9v8m for pod openshift-network-diagnostics/network-check-target-g5hnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 10:57:34.318052 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:34.317642    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2830f09-58f3-425c-a831-e5e34b542e78-kube-api-access-n9v8m podName:f2830f09-58f3-425c-a831-e5e34b542e78 nodeName:}" failed. No retries permitted until 2026-07-17 10:57:42.317623647 +0000 UTC m=+17.247163285 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-n9v8m" (UniqueName: "kubernetes.io/projected/f2830f09-58f3-425c-a831-e5e34b542e78-kube-api-access-n9v8m") pod "network-check-target-g5hnv" (UID: "f2830f09-58f3-425c-a831-e5e34b542e78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 10:57:34.660182 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:34.660146    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:57:34.660615 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:34.660147    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:34.660615 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:34.660292    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5hnv" podUID="f2830f09-58f3-425c-a831-e5e34b542e78"
Jul 17 10:57:34.660615 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:34.660394    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7fx4" podUID="84ae7bd0-6126-491f-9126-2d64506af889"
Jul 17 10:57:35.661408 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:35.661347    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:35.661910 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:35.661486    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c9b" podUID="e9c3cb80-365e-4c10-bde0-1e9f17948a97"
Jul 17 10:57:36.660799 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:36.660128    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:57:36.660799 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:36.660255    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5hnv" podUID="f2830f09-58f3-425c-a831-e5e34b542e78"
Jul 17 10:57:36.660799 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:36.660665    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:36.660799 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:36.660753    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7fx4" podUID="84ae7bd0-6126-491f-9126-2d64506af889"
Jul 17 10:57:37.659930 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:37.659894    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:37.660415 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:37.660040    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c9b" podUID="e9c3cb80-365e-4c10-bde0-1e9f17948a97"
Jul 17 10:57:38.659708 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:38.659667    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:57:38.659861 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:38.659667    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:38.659861 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:38.659802    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5hnv" podUID="f2830f09-58f3-425c-a831-e5e34b542e78"
Jul 17 10:57:38.659861 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:38.659838    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7fx4" podUID="84ae7bd0-6126-491f-9126-2d64506af889"
Jul 17 10:57:39.660081 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:39.660030    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:39.660549 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:39.660195    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c9b" podUID="e9c3cb80-365e-4c10-bde0-1e9f17948a97"
Jul 17 10:57:40.659930 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:40.659893    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:57:40.660186 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:40.659898    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:40.660186 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:40.660033    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5hnv" podUID="f2830f09-58f3-425c-a831-e5e34b542e78"
Jul 17 10:57:40.660186 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:40.660126    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7fx4" podUID="84ae7bd0-6126-491f-9126-2d64506af889"
Jul 17 10:57:41.659669 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:41.659635    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:41.659833 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:41.659782    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c9b" podUID="e9c3cb80-365e-4c10-bde0-1e9f17948a97"
Jul 17 10:57:42.173701 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:42.173669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9c3cb80-365e-4c10-bde0-1e9f17948a97-metrics-certs\") pod \"network-metrics-daemon-s5c9b\" (UID: \"e9c3cb80-365e-4c10-bde0-1e9f17948a97\") " pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:42.174216 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:42.173880    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 10:57:42.174216 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:42.173978    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9c3cb80-365e-4c10-bde0-1e9f17948a97-metrics-certs podName:e9c3cb80-365e-4c10-bde0-1e9f17948a97 nodeName:}" failed. No retries permitted until 2026-07-17 10:57:58.173955801 +0000 UTC m=+33.103495429 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9c3cb80-365e-4c10-bde0-1e9f17948a97-metrics-certs") pod "network-metrics-daemon-s5c9b" (UID: "e9c3cb80-365e-4c10-bde0-1e9f17948a97") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 10:57:42.275122 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:42.275080    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84ae7bd0-6126-491f-9126-2d64506af889-original-pull-secret\") pod \"global-pull-secret-syncer-b7fx4\" (UID: \"84ae7bd0-6126-491f-9126-2d64506af889\") " pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:42.275306 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:42.275190    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 10:57:42.275306 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:42.275259    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84ae7bd0-6126-491f-9126-2d64506af889-original-pull-secret podName:84ae7bd0-6126-491f-9126-2d64506af889 nodeName:}" failed. No retries permitted until 2026-07-17 10:57:58.275243262 +0000 UTC m=+33.204782906 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84ae7bd0-6126-491f-9126-2d64506af889-original-pull-secret") pod "global-pull-secret-syncer-b7fx4" (UID: "84ae7bd0-6126-491f-9126-2d64506af889") : object "kube-system"/"original-pull-secret" not registered
Jul 17 10:57:42.375684 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:42.375651    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9v8m\" (UniqueName: \"kubernetes.io/projected/f2830f09-58f3-425c-a831-e5e34b542e78-kube-api-access-n9v8m\") pod \"network-check-target-g5hnv\" (UID: \"f2830f09-58f3-425c-a831-e5e34b542e78\") " pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:57:42.375857 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:42.375790    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 10:57:42.375857 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:42.375809    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 10:57:42.375857 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:42.375821    2563 projected.go:196] Error preparing data for projected volume kube-api-access-n9v8m for pod openshift-network-diagnostics/network-check-target-g5hnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 10:57:42.375996 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:42.375873    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2830f09-58f3-425c-a831-e5e34b542e78-kube-api-access-n9v8m podName:f2830f09-58f3-425c-a831-e5e34b542e78 nodeName:}" failed. No retries permitted until 2026-07-17 10:57:58.375861177 +0000 UTC m=+33.305400799 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-n9v8m" (UniqueName: "kubernetes.io/projected/f2830f09-58f3-425c-a831-e5e34b542e78-kube-api-access-n9v8m") pod "network-check-target-g5hnv" (UID: "f2830f09-58f3-425c-a831-e5e34b542e78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 10:57:42.659519 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:42.659480    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:57:42.659690 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:42.659494    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:42.660088 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:42.660017    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7fx4" podUID="84ae7bd0-6126-491f-9126-2d64506af889"
Jul 17 10:57:42.660738 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:42.660701    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5hnv" podUID="f2830f09-58f3-425c-a831-e5e34b542e78"
Jul 17 10:57:43.659504 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:43.659461    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:43.659976 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:43.659611    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c9b" podUID="e9c3cb80-365e-4c10-bde0-1e9f17948a97"
Jul 17 10:57:44.660211 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:44.660174    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:57:44.660633 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:44.660288    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5hnv" podUID="f2830f09-58f3-425c-a831-e5e34b542e78"
Jul 17 10:57:44.660633 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:44.660351    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:44.660633 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:44.660479    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7fx4" podUID="84ae7bd0-6126-491f-9126-2d64506af889"
Jul 17 10:57:45.660187 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:45.660157    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:45.660354 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:45.660248    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c9b" podUID="e9c3cb80-365e-4c10-bde0-1e9f17948a97"
Jul 17 10:57:46.659561 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:46.659524    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:46.659758 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:46.659697    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7fx4" podUID="84ae7bd0-6126-491f-9126-2d64506af889"
Jul 17 10:57:46.659758 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:46.659711    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:57:46.659880 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:46.659836    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5hnv" podUID="f2830f09-58f3-425c-a831-e5e34b542e78"
Jul 17 10:57:47.659493 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:47.659337    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:47.659811 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:47.659579    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c9b" podUID="e9c3cb80-365e-4c10-bde0-1e9f17948a97"
Jul 17 10:57:47.857987 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:47.857945    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l426v" event={"ID":"609409ce-006b-440e-8403-dda1bef2dd5b","Type":"ContainerStarted","Data":"710c683ecf7c01a7b25e76e41069cfcaeab1ac6053f7f6a466c2b50f787d85c9"}
Jul 17 10:57:47.859749 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:47.859598    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j4mp9" event={"ID":"60837033-c957-4341-8f68-9e70bf72cce6","Type":"ContainerStarted","Data":"c8b1788f7bd35fed1126e3702396da76e73aa8189dbca2cf021e54a4fd09f01e"}
Jul 17 10:57:47.861319 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:47.861298    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-233.ec2.internal" event={"ID":"098d9536b1eb81f9184c22a2bb857164","Type":"ContainerStarted","Data":"7f5aeb3362ec66b9c049aa61fe9b51c87c56998479f5789f11d2593ff366841a"}
Jul 17 10:57:47.875614 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:47.875567    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-l426v" podStartSLOduration=1.359283296 podStartE2EDuration="21.875550075s" totalImagesPullingTime="20.516266779s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 10:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:57:47.8752828 +0000 UTC m=+22.804822441" watchObservedRunningTime="2026-07-17 10:57:47.875550075 +0000 UTC m=+22.805089771"
Jul 17 10:57:47.893258 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:47.893217    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-233.ec2.internal" podStartSLOduration=21.893202908 podStartE2EDuration="21.893202908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 10:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:57:47.892683353 +0000 UTC m=+22.822222994" watchObservedRunningTime="2026-07-17 10:57:47.893202908 +0000 UTC m=+22.822742549"
Jul 17 10:57:47.911355 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:47.911272    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-j4mp9" podStartSLOduration=1.29912507 podStartE2EDuration="21.91125687s" totalImagesPullingTime="20.6121318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 10:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:57:47.910776616 +0000 UTC m=+22.840316258" watchObservedRunningTime="2026-07-17 10:57:47.91125687 +0000 UTC m=+22.840796510"
Jul 17 10:57:48.659883 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:48.659691    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:57:48.660656 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:48.659697    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:48.660656 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:48.659969    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5hnv" podUID="f2830f09-58f3-425c-a831-e5e34b542e78"
Jul 17 10:57:48.660656 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:48.660023    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7fx4" podUID="84ae7bd0-6126-491f-9126-2d64506af889"
Jul 17 10:57:48.864675 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:48.864409    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-87r57" event={"ID":"f1f509b8-7e29-445b-ad25-dd54a1c6dc07","Type":"ContainerStarted","Data":"6972ac4fdad4782a11c6ba58222e6af3290f586940e921ee19c0533d6c313008"}
Jul 17 10:57:48.865858 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:48.865834    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59" event={"ID":"fc69e11f-7c22-4067-8169-886d341a4b70","Type":"ContainerStarted","Data":"4b1ea8f1c64aa8935cb6f143b163527766ddeef78aa23033a9fc35a3a1195dea"}
Jul 17 10:57:48.867325 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:48.867297    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kz6gd" event={"ID":"73402c16-36b4-442c-9009-4d41bd9003fc","Type":"ContainerStarted","Data":"48ea04774fb7fc22c7680407276dc7ca7d14831a76f5bee1ec0df4648b5eebde"}
Jul 17 10:57:48.868947 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:48.868925    2563 generic.go:356] "Generic (PLEG): container finished" podID="4b3498a1-e561-4239-8878-ad9595ad312a" containerID="daf5073aeff51760404f484019ba8b0bd085f93fe6bd6ad9b02d737fabbbd93e" exitCode=0
Jul 17 10:57:48.869054 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:48.868956    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcg7j" event={"ID":"4b3498a1-e561-4239-8878-ad9595ad312a","Type":"ContainerDied","Data":"daf5073aeff51760404f484019ba8b0bd085f93fe6bd6ad9b02d737fabbbd93e"}
Jul 17 10:57:48.870842 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:48.870791    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qqqf9" event={"ID":"72f5631e-c70d-47b8-9405-d8b986fd12e7","Type":"ContainerStarted","Data":"75182f6371bd32417ee9d89e7f1203a7c06ac2c961443acdc984c12345020ab9"}
Jul 17 10:57:48.872645 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:48.872622    2563 generic.go:356] "Generic (PLEG): container finished" podID="d2c42ae2a6854f18bdedf80ad478b60e" containerID="462d2ae034c1246e17dd33431e287d4dd4ab1ba2cff8638fe7294ade70cb1af1" exitCode=0
Jul 17 10:57:48.872748 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:48.872698    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-233.ec2.internal" event={"ID":"d2c42ae2a6854f18bdedf80ad478b60e","Type":"ContainerDied","Data":"462d2ae034c1246e17dd33431e287d4dd4ab1ba2cff8638fe7294ade70cb1af1"}
Jul 17 10:57:48.875741 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:48.875724    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nj5mj_55d1de37-8635-4de8-ab59-4ebfd0346a40/ovn-acl-logging/0.log"
Jul 17 10:57:48.876152 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:48.876107    2563 generic.go:356] "Generic (PLEG): container finished" podID="55d1de37-8635-4de8-ab59-4ebfd0346a40" containerID="6cb4f3132a1dbe4d131fbeca791114998b4aa6b6d41536cf63a3d34c9d0d2309" exitCode=1
Jul 17 10:57:48.876262 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:48.876183    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj" event={"ID":"55d1de37-8635-4de8-ab59-4ebfd0346a40","Type":"ContainerStarted","Data":"d6c9223aa4f9126585ee6fa1056a748a924b9eafba0c186fd87556148d8280fa"}
Jul 17 10:57:48.876320 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:48.876263    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj" event={"ID":"55d1de37-8635-4de8-ab59-4ebfd0346a40","Type":"ContainerStarted","Data":"c6211a408c52c6ac67522f7b521f228a97a1a08ef4e094f9288f5352ad282333"}
Jul 17 10:57:48.876376 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:48.876318    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj" event={"ID":"55d1de37-8635-4de8-ab59-4ebfd0346a40","Type":"ContainerStarted","Data":"79c77896239a61eeed550e806f5a06668a1723291e6363dd691d9451bed9b4f4"}
Jul 17 10:57:48.876376 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:48.876365    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj" event={"ID":"55d1de37-8635-4de8-ab59-4ebfd0346a40","Type":"ContainerStarted","Data":"e472ec2bfbe368544c208821861496ebb74e18c5710159a4b20c846c659deb3a"}
Jul 17 10:57:48.876479 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:48.876411    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj" event={"ID":"55d1de37-8635-4de8-ab59-4ebfd0346a40","Type":"ContainerDied","Data":"6cb4f3132a1dbe4d131fbeca791114998b4aa6b6d41536cf63a3d34c9d0d2309"}
Jul 17 10:57:48.876479 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:48.876460    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj" event={"ID":"55d1de37-8635-4de8-ab59-4ebfd0346a40","Type":"ContainerStarted","Data":"d81d8374cb9306597d187497f2ed8c6f0a44712ce2e5c7009b9cb2c53247a5a9"}
Jul 17 10:57:48.877785 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:48.877765    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pvvr8" event={"ID":"d2304352-6b6e-42d3-af75-146ba8b85e0b","Type":"ContainerStarted","Data":"73ae53d7f4eb525c2c35bd92443ed55e35b921057a7c0aa6d90ea7232b070e96"}
Jul 17 10:57:48.885340 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:48.885302    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-87r57" podStartSLOduration=2.456782715 podStartE2EDuration="22.885285216s" totalImagesPullingTime="20.428502501s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 10:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:57:48.885256474 +0000 UTC m=+23.814796119" watchObservedRunningTime="2026-07-17 10:57:48.885285216 +0000 UTC m=+23.814824859"
Jul 17 10:57:48.935397 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:48.933978    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kz6gd" podStartSLOduration=11.000320167 podStartE2EDuration="22.933958733s" totalImagesPullingTime="11.933638566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:57:48.933764534 +0000 UTC m=+23.863304177" watchObservedRunningTime="2026-07-17 10:57:48.933958733 +0000 UTC m=+23.863498377"
Jul 17 10:57:48.959501 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:48.959453    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pvvr8" podStartSLOduration=2.410965515 podStartE2EDuration="22.959440116s" totalImagesPullingTime="20.548474601s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 10:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:57:48.959168783 +0000 UTC m=+23.888708423" watchObservedRunningTime="2026-07-17 10:57:48.959440116 +0000 UTC m=+23.888979768"
Jul 17 10:57:49.004365 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:49.004304    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qqqf9" podStartSLOduration=2.396215046 podStartE2EDuration="23.004281714s" totalImagesPullingTime="20.608066668s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 10:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:57:49.003723276 +0000 UTC m=+23.933262918" watchObservedRunningTime="2026-07-17 10:57:49.004281714 +0000 UTC m=+23.933821562"
Jul 17 10:57:49.134885 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:49.134860    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:57:49.221634 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:49.221604    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:57:49.293726 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:49.293696    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:57:49.610676 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:49.610644    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:57:49.659950 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:49.659924    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:49.660354 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:49.660045    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c9b" podUID="e9c3cb80-365e-4c10-bde0-1e9f17948a97"
Jul 17 10:57:49.681638 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:49.681600    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 17 10:57:49.710508 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:49.710480    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:57:49.882100 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:49.882030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59" event={"ID":"fc69e11f-7c22-4067-8169-886d341a4b70","Type":"ContainerStarted","Data":"d82f896a24512bb210244285595f40ad05c71817808e19aadfb4e0d61a0bb10c"}
Jul 17 10:57:49.917657 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:49.917609    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kz6gd"
Jul 17 10:57:49.918369 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:49.918349    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kz6gd"
Jul 17 10:57:50.441223 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:50.441143    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:57:50.566269 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:50.566176    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T10:57:49.681618731Z","UUID":"3df0e00d-e55b-4c37-b991-ee223ab8973a","Handler":null,"Name":"","Endpoint":""}
Jul 17 10:57:50.568116 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:50.568092    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 17 10:57:50.568263 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:50.568154    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 17 10:57:50.659696 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:50.659661    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:50.659696 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:50.659679    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:57:50.659984 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:50.659777    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7fx4" podUID="84ae7bd0-6126-491f-9126-2d64506af889"
Jul 17 10:57:50.659984 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:50.659897    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5hnv" podUID="f2830f09-58f3-425c-a831-e5e34b542e78"
Jul 17 10:57:50.846661 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:50.846626    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:57:50.886509 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:50.886062    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-233.ec2.internal" event={"ID":"d2c42ae2a6854f18bdedf80ad478b60e","Type":"ContainerStarted","Data":"d099ab0e6d93ece88305bd8aeff57b399090ca18f7d87dd38e859a575b962a08"}
Jul 17 10:57:50.886509 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:50.886475    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kz6gd"
Jul 17 10:57:50.890159 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:50.887697    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kz6gd"
Jul 17 10:57:50.915842 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:50.915790    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-233.ec2.internal" podStartSLOduration=24.915771946 podStartE2EDuration="24.915771946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 10:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:57:50.915270324 +0000 UTC m=+25.844809964" watchObservedRunningTime="2026-07-17 10:57:50.915771946 +0000 UTC m=+25.845311588"
Jul 17 10:57:50.965391 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:50.965312    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:57:51.155295 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:51.155258    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:57:51.660017 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:51.659987    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:51.660535 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:51.660148    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c9b" podUID="e9c3cb80-365e-4c10-bde0-1e9f17948a97"
Jul 17 10:57:51.890803 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:51.890768    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59" event={"ID":"fc69e11f-7c22-4067-8169-886d341a4b70","Type":"ContainerStarted","Data":"1cb735cb0d4b33d4b2e23e0f9159b665cb4641602429778673b0a248b000aa6f"}
Jul 17 10:57:51.894358 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:51.894335    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nj5mj_55d1de37-8635-4de8-ab59-4ebfd0346a40/ovn-acl-logging/0.log"
Jul 17 10:57:51.894785 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:51.894759    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj" event={"ID":"55d1de37-8635-4de8-ab59-4ebfd0346a40","Type":"ContainerStarted","Data":"27f93144096376b934b1e0dd6d0991c8ea8877637963e4fb72bb3f8eab62889a"}
Jul 17 10:57:51.919104 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:51.918997    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p9h59" podStartSLOduration=3.099592432 podStartE2EDuration="25.918978084s" totalImagesPullingTime="22.819385652s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 10:57:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:57:51.918943204 +0000 UTC m=+26.848482849" watchObservedRunningTime="2026-07-17 10:57:51.918978084 +0000 UTC m=+26.848517726"
Jul 17 10:57:52.660170 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:52.660137    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:57:52.660567 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:52.660247    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5hnv" podUID="f2830f09-58f3-425c-a831-e5e34b542e78"
Jul 17 10:57:52.660567 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:52.660290    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:52.660567 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:52.660417    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7fx4" podUID="84ae7bd0-6126-491f-9126-2d64506af889"
Jul 17 10:57:53.659881 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:53.659719    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:53.660103 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:53.659966    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c9b" podUID="e9c3cb80-365e-4c10-bde0-1e9f17948a97"
Jul 17 10:57:53.900228 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:53.900190    2563 generic.go:356] "Generic (PLEG): container finished" podID="4b3498a1-e561-4239-8878-ad9595ad312a" containerID="651c50b553afe5bab224c2386a1be711010fb1480c476748db2a3ae4ef056463" exitCode=0
Jul 17 10:57:53.901038 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:53.900277    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcg7j" event={"ID":"4b3498a1-e561-4239-8878-ad9595ad312a","Type":"ContainerDied","Data":"651c50b553afe5bab224c2386a1be711010fb1480c476748db2a3ae4ef056463"}
Jul 17 10:57:53.907458 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:53.907439    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nj5mj_55d1de37-8635-4de8-ab59-4ebfd0346a40/ovn-acl-logging/0.log"
Jul 17 10:57:53.908598 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:53.907910    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj" event={"ID":"55d1de37-8635-4de8-ab59-4ebfd0346a40","Type":"ContainerStarted","Data":"2b749866af71b5dbd8e924695879d12e4fbd13c9ee372e01c5316377d33e6a91"}
Jul 17 10:57:53.908716 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:53.908627    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:53.909117 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:53.909103    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:53.909215 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:53.909166    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:53.909273 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:53.909235    2563 scope.go:117] "RemoveContainer" containerID="6cb4f3132a1dbe4d131fbeca791114998b4aa6b6d41536cf63a3d34c9d0d2309"
Jul 17 10:57:53.924296 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:53.924277    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:53.925286 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:53.925271    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:57:54.659581 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:54.659549    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:57:54.659581 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:54.659579    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:54.659921 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:54.659665    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5hnv" podUID="f2830f09-58f3-425c-a831-e5e34b542e78"
Jul 17 10:57:54.659921 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:54.659826    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7fx4" podUID="84ae7bd0-6126-491f-9126-2d64506af889"
Jul 17 10:57:54.913339 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:54.913290    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nj5mj_55d1de37-8635-4de8-ab59-4ebfd0346a40/ovn-acl-logging/0.log"
Jul 17 10:57:54.913646 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:54.913622    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj" event={"ID":"55d1de37-8635-4de8-ab59-4ebfd0346a40","Type":"ContainerStarted","Data":"24e5caeb138813b10a28257134b879d145d9a7906d7bdf8afc4c28520c55098f"}
Jul 17 10:57:54.947043 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:54.947000    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj" podStartSLOduration=7.8600904499999995 podStartE2EDuration="28.946987587s" totalImagesPullingTime="21.086897137s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 10:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:57:54.946446643 +0000 UTC m=+29.875986284" watchObservedRunningTime="2026-07-17 10:57:54.946987587 +0000 UTC m=+29.876527227"
Jul 17 10:57:55.470633 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:55.470463    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g5hnv"]
Jul 17 10:57:55.470801 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:55.470699    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:57:55.470801 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:55.470776    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5hnv" podUID="f2830f09-58f3-425c-a831-e5e34b542e78"
Jul 17 10:57:55.473321 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:55.473301    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b7fx4"]
Jul 17 10:57:55.473448 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:55.473378    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:55.473497 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:55.473457    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7fx4" podUID="84ae7bd0-6126-491f-9126-2d64506af889"
Jul 17 10:57:55.473851 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:55.473829    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s5c9b"]
Jul 17 10:57:55.473953 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:55.473943    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:55.474078 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:55.474044    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c9b" podUID="e9c3cb80-365e-4c10-bde0-1e9f17948a97"
Jul 17 10:57:55.917194 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:55.917151    2563 generic.go:356] "Generic (PLEG): container finished" podID="4b3498a1-e561-4239-8878-ad9595ad312a" containerID="0fd79a941552a0688235240e98d1831815f1d8c579f289939258821721f6e664" exitCode=0
Jul 17 10:57:55.917651 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:55.917231    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcg7j" event={"ID":"4b3498a1-e561-4239-8878-ad9595ad312a","Type":"ContainerDied","Data":"0fd79a941552a0688235240e98d1831815f1d8c579f289939258821721f6e664"}
Jul 17 10:57:57.659758 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:57.659730    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:57.660165 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:57.659734    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:57.660165 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:57.659836    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7fx4" podUID="84ae7bd0-6126-491f-9126-2d64506af889"
Jul 17 10:57:57.660165 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:57.659859    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:57:57.660165 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:57.659925    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c9b" podUID="e9c3cb80-365e-4c10-bde0-1e9f17948a97"
Jul 17 10:57:57.660165 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:57.659969    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5hnv" podUID="f2830f09-58f3-425c-a831-e5e34b542e78"
Jul 17 10:57:57.924047 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:57.924015    2563 generic.go:356] "Generic (PLEG): container finished" podID="4b3498a1-e561-4239-8878-ad9595ad312a" containerID="69e5998742026cd0ad778798ecc0b5f1a0018bd7c5f41de302e1251386fa19f4" exitCode=0
Jul 17 10:57:57.924207 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:57.924107    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcg7j" event={"ID":"4b3498a1-e561-4239-8878-ad9595ad312a","Type":"ContainerDied","Data":"69e5998742026cd0ad778798ecc0b5f1a0018bd7c5f41de302e1251386fa19f4"}
Jul 17 10:57:58.190252 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:58.190178    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9c3cb80-365e-4c10-bde0-1e9f17948a97-metrics-certs\") pod \"network-metrics-daemon-s5c9b\" (UID: \"e9c3cb80-365e-4c10-bde0-1e9f17948a97\") " pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:58.190402 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:58.190319    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 10:57:58.190447 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:58.190435    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9c3cb80-365e-4c10-bde0-1e9f17948a97-metrics-certs podName:e9c3cb80-365e-4c10-bde0-1e9f17948a97 nodeName:}" failed. No retries permitted until 2026-07-17 10:58:30.190413827 +0000 UTC m=+65.119953450 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9c3cb80-365e-4c10-bde0-1e9f17948a97-metrics-certs") pod "network-metrics-daemon-s5c9b" (UID: "e9c3cb80-365e-4c10-bde0-1e9f17948a97") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 10:57:58.291245 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:58.291210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84ae7bd0-6126-491f-9126-2d64506af889-original-pull-secret\") pod \"global-pull-secret-syncer-b7fx4\" (UID: \"84ae7bd0-6126-491f-9126-2d64506af889\") " pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:58.291396 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:58.291311    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 10:57:58.291396 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:58.291359    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84ae7bd0-6126-491f-9126-2d64506af889-original-pull-secret podName:84ae7bd0-6126-491f-9126-2d64506af889 nodeName:}" failed. No retries permitted until 2026-07-17 10:58:30.291346364 +0000 UTC m=+65.220885982 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84ae7bd0-6126-491f-9126-2d64506af889-original-pull-secret") pod "global-pull-secret-syncer-b7fx4" (UID: "84ae7bd0-6126-491f-9126-2d64506af889") : object "kube-system"/"original-pull-secret" not registered
Jul 17 10:57:58.392141 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:58.392109    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9v8m\" (UniqueName: \"kubernetes.io/projected/f2830f09-58f3-425c-a831-e5e34b542e78-kube-api-access-n9v8m\") pod \"network-check-target-g5hnv\" (UID: \"f2830f09-58f3-425c-a831-e5e34b542e78\") " pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:57:58.392294 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:58.392231    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 10:57:58.392294 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:58.392245    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 10:57:58.392294 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:58.392253    2563 projected.go:196] Error preparing data for projected volume kube-api-access-n9v8m for pod openshift-network-diagnostics/network-check-target-g5hnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 10:57:58.392393 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:58.392298    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2830f09-58f3-425c-a831-e5e34b542e78-kube-api-access-n9v8m podName:f2830f09-58f3-425c-a831-e5e34b542e78 nodeName:}" failed. No retries permitted until 2026-07-17 10:58:30.392286377 +0000 UTC m=+65.321825996 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-n9v8m" (UniqueName: "kubernetes.io/projected/f2830f09-58f3-425c-a831-e5e34b542e78-kube-api-access-n9v8m") pod "network-check-target-g5hnv" (UID: "f2830f09-58f3-425c-a831-e5e34b542e78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 10:57:58.978476 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:58.978443    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:57:59.041146 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:59.041108    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:57:59.659755 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:59.659698    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:57:59.659755 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:59.659727    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:57:59.659755 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:57:59.659698    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:57:59.660008 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:59.659848    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5hnv" podUID="f2830f09-58f3-425c-a831-e5e34b542e78"
Jul 17 10:57:59.660008 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:59.659883    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7fx4" podUID="84ae7bd0-6126-491f-9126-2d64506af889"
Jul 17 10:57:59.660008 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:57:59.659958    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c9b" podUID="e9c3cb80-365e-4c10-bde0-1e9f17948a97"
Jul 17 10:58:00.298772 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.298594    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:00.379835 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.379811    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-233.ec2.internal" event="NodeReady"
Jul 17 10:58:00.380009 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.379960    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 17 10:58:00.403558 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.403520    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:00.428761 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.428731    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-h4cnn"]
Jul 17 10:58:00.430814 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.430797    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h4cnn"
Jul 17 10:58:00.433351 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.433326    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w544b\""
Jul 17 10:58:00.433474 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.433336    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 17 10:58:00.433554 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.433491    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 17 10:58:00.435254 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.435225    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gzhwh"]
Jul 17 10:58:00.437038 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.437016    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gzhwh"
Jul 17 10:58:00.439576 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.439426    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 17 10:58:00.439576 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.439469    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 17 10:58:00.439576 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.439494    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9lr9m\""
Jul 17 10:58:00.439725 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.439634    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 17 10:58:00.442735 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.442683    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h4cnn"]
Jul 17 10:58:00.448715 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.448697    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gzhwh"]
Jul 17 10:58:00.508387 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.508353    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eced893c-6854-4d8c-bbc2-6d6cfa602028-cert\") pod \"ingress-canary-gzhwh\" (UID: \"eced893c-6854-4d8c-bbc2-6d6cfa602028\") " pod="openshift-ingress-canary/ingress-canary-gzhwh"
Jul 17 10:58:00.508566 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.508403    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6nkj\" (UniqueName: \"kubernetes.io/projected/bef28d6a-f451-4c22-adca-7179590804ea-kube-api-access-q6nkj\") pod \"dns-default-h4cnn\" (UID: \"bef28d6a-f451-4c22-adca-7179590804ea\") " pod="openshift-dns/dns-default-h4cnn"
Jul 17 10:58:00.508566 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.508426    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef28d6a-f451-4c22-adca-7179590804ea-tmp-dir\") pod \"dns-default-h4cnn\" (UID: \"bef28d6a-f451-4c22-adca-7179590804ea\") " pod="openshift-dns/dns-default-h4cnn"
Jul 17 10:58:00.508566 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.508506    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htmlm\" (UniqueName: \"kubernetes.io/projected/eced893c-6854-4d8c-bbc2-6d6cfa602028-kube-api-access-htmlm\") pod \"ingress-canary-gzhwh\" (UID: \"eced893c-6854-4d8c-bbc2-6d6cfa602028\") " pod="openshift-ingress-canary/ingress-canary-gzhwh"
Jul 17 10:58:00.508566 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.508545    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bef28d6a-f451-4c22-adca-7179590804ea-config-volume\") pod \"dns-default-h4cnn\" (UID: \"bef28d6a-f451-4c22-adca-7179590804ea\") " pod="openshift-dns/dns-default-h4cnn"
Jul 17 10:58:00.508725 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.508592    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef28d6a-f451-4c22-adca-7179590804ea-metrics-tls\") pod \"dns-default-h4cnn\" (UID: \"bef28d6a-f451-4c22-adca-7179590804ea\") " pod="openshift-dns/dns-default-h4cnn"
Jul 17 10:58:00.576511 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.576434    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:00.609476 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.609443    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6nkj\" (UniqueName: \"kubernetes.io/projected/bef28d6a-f451-4c22-adca-7179590804ea-kube-api-access-q6nkj\") pod \"dns-default-h4cnn\" (UID: \"bef28d6a-f451-4c22-adca-7179590804ea\") " pod="openshift-dns/dns-default-h4cnn"
Jul 17 10:58:00.609648 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.609489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef28d6a-f451-4c22-adca-7179590804ea-tmp-dir\") pod \"dns-default-h4cnn\" (UID: \"bef28d6a-f451-4c22-adca-7179590804ea\") " pod="openshift-dns/dns-default-h4cnn"
Jul 17 10:58:00.609648 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.609626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htmlm\" (UniqueName: \"kubernetes.io/projected/eced893c-6854-4d8c-bbc2-6d6cfa602028-kube-api-access-htmlm\") pod \"ingress-canary-gzhwh\" (UID: \"eced893c-6854-4d8c-bbc2-6d6cfa602028\") " pod="openshift-ingress-canary/ingress-canary-gzhwh"
Jul 17 10:58:00.609756 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.609694    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bef28d6a-f451-4c22-adca-7179590804ea-config-volume\") pod \"dns-default-h4cnn\" (UID: \"bef28d6a-f451-4c22-adca-7179590804ea\") " pod="openshift-dns/dns-default-h4cnn"
Jul 17 10:58:00.609756 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.609719    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef28d6a-f451-4c22-adca-7179590804ea-metrics-tls\") pod \"dns-default-h4cnn\" (UID: \"bef28d6a-f451-4c22-adca-7179590804ea\") " pod="openshift-dns/dns-default-h4cnn"
Jul 17 10:58:00.609842 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.609754    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eced893c-6854-4d8c-bbc2-6d6cfa602028-cert\") pod \"ingress-canary-gzhwh\" (UID: \"eced893c-6854-4d8c-bbc2-6d6cfa602028\") " pod="openshift-ingress-canary/ingress-canary-gzhwh"
Jul 17 10:58:00.610281 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:00.610025    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 10:58:00.610281 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:00.610115    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef28d6a-f451-4c22-adca-7179590804ea-metrics-tls podName:bef28d6a-f451-4c22-adca-7179590804ea nodeName:}" failed. No retries permitted until 2026-07-17 10:58:01.110095589 +0000 UTC m=+36.039635225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bef28d6a-f451-4c22-adca-7179590804ea-metrics-tls") pod "dns-default-h4cnn" (UID: "bef28d6a-f451-4c22-adca-7179590804ea") : secret "dns-default-metrics-tls" not found
Jul 17 10:58:00.610281 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.610152    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef28d6a-f451-4c22-adca-7179590804ea-tmp-dir\") pod \"dns-default-h4cnn\" (UID: \"bef28d6a-f451-4c22-adca-7179590804ea\") " pod="openshift-dns/dns-default-h4cnn"
Jul 17 10:58:00.610498 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.610443    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bef28d6a-f451-4c22-adca-7179590804ea-config-volume\") pod \"dns-default-h4cnn\" (UID: \"bef28d6a-f451-4c22-adca-7179590804ea\") " pod="openshift-dns/dns-default-h4cnn"
Jul 17 10:58:00.614536 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.614517    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eced893c-6854-4d8c-bbc2-6d6cfa602028-cert\") pod \"ingress-canary-gzhwh\" (UID: \"eced893c-6854-4d8c-bbc2-6d6cfa602028\") " pod="openshift-ingress-canary/ingress-canary-gzhwh"
Jul 17 10:58:00.622204 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.622182    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6nkj\" (UniqueName: \"kubernetes.io/projected/bef28d6a-f451-4c22-adca-7179590804ea-kube-api-access-q6nkj\") pod \"dns-default-h4cnn\" (UID: \"bef28d6a-f451-4c22-adca-7179590804ea\") " pod="openshift-dns/dns-default-h4cnn"
Jul 17 10:58:00.622308 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.622263    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htmlm\" (UniqueName: \"kubernetes.io/projected/eced893c-6854-4d8c-bbc2-6d6cfa602028-kube-api-access-htmlm\") pod \"ingress-canary-gzhwh\" (UID: \"eced893c-6854-4d8c-bbc2-6d6cfa602028\") " pod="openshift-ingress-canary/ingress-canary-gzhwh"
Jul 17 10:58:00.677972 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.677899    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:00.750084 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:00.750042    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gzhwh"
Jul 17 10:58:01.113434 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:01.113396    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef28d6a-f451-4c22-adca-7179590804ea-metrics-tls\") pod \"dns-default-h4cnn\" (UID: \"bef28d6a-f451-4c22-adca-7179590804ea\") " pod="openshift-dns/dns-default-h4cnn"
Jul 17 10:58:01.113608 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:01.113527    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 10:58:01.113608 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:01.113589    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef28d6a-f451-4c22-adca-7179590804ea-metrics-tls podName:bef28d6a-f451-4c22-adca-7179590804ea nodeName:}" failed. No retries permitted until 2026-07-17 10:58:02.113572565 +0000 UTC m=+37.043112193 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bef28d6a-f451-4c22-adca-7179590804ea-metrics-tls") pod "dns-default-h4cnn" (UID: "bef28d6a-f451-4c22-adca-7179590804ea") : secret "dns-default-metrics-tls" not found
Jul 17 10:58:01.659696 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:01.659659    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:58:01.660164 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:01.659660    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:58:01.660164 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:01.659660    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:58:01.664099 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:01.663698    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 10:58:01.664099 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:01.663756    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bvrbg\""
Jul 17 10:58:01.664099 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:01.663698    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 10:58:01.664099 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:01.663974    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m4ccj\""
Jul 17 10:58:01.664099 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:01.664001    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 10:58:01.664424 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:01.664161    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 10:58:02.120874 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:02.120788    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef28d6a-f451-4c22-adca-7179590804ea-metrics-tls\") pod \"dns-default-h4cnn\" (UID: \"bef28d6a-f451-4c22-adca-7179590804ea\") " pod="openshift-dns/dns-default-h4cnn"
Jul 17 10:58:02.121047 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:02.120963    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 10:58:02.121047 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:02.121041    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef28d6a-f451-4c22-adca-7179590804ea-metrics-tls podName:bef28d6a-f451-4c22-adca-7179590804ea nodeName:}" failed. No retries permitted until 2026-07-17 10:58:04.121020454 +0000 UTC m=+39.050560087 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bef28d6a-f451-4c22-adca-7179590804ea-metrics-tls") pod "dns-default-h4cnn" (UID: "bef28d6a-f451-4c22-adca-7179590804ea") : secret "dns-default-metrics-tls" not found
Jul 17 10:58:02.354218 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:02.354190    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gzhwh"]
Jul 17 10:58:02.357586 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:58:02.357561    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeced893c_6854_4d8c_bbc2_6d6cfa602028.slice/crio-3205e30ea484851d8baabeae1148af9bc5d48f726af1fe112b9a82d82720d9a2 WatchSource:0}: Error finding container 3205e30ea484851d8baabeae1148af9bc5d48f726af1fe112b9a82d82720d9a2: Status 404 returned error can't find the container with id 3205e30ea484851d8baabeae1148af9bc5d48f726af1fe112b9a82d82720d9a2
Jul 17 10:58:02.936577 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:02.936539    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gzhwh" event={"ID":"eced893c-6854-4d8c-bbc2-6d6cfa602028","Type":"ContainerStarted","Data":"3205e30ea484851d8baabeae1148af9bc5d48f726af1fe112b9a82d82720d9a2"}
Jul 17 10:58:02.939359 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:02.939150    2563 generic.go:356] "Generic (PLEG): container finished" podID="4b3498a1-e561-4239-8878-ad9595ad312a" containerID="4caf6290d2ff4044c840f59a7bd5744304c8f6f7a02cb892b4b41df8ad3daf84" exitCode=0
Jul 17 10:58:02.939515 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:02.939224    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcg7j" event={"ID":"4b3498a1-e561-4239-8878-ad9595ad312a","Type":"ContainerDied","Data":"4caf6290d2ff4044c840f59a7bd5744304c8f6f7a02cb892b4b41df8ad3daf84"}
Jul 17 10:58:03.945713 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:03.945444    2563 generic.go:356] "Generic (PLEG): container finished" podID="4b3498a1-e561-4239-8878-ad9595ad312a" containerID="4df41a9140b8e660da049b54685a1d467e0ed205ace27ba9d47d6f689fd019ea" exitCode=0
Jul 17 10:58:03.945713 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:03.945513    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcg7j" event={"ID":"4b3498a1-e561-4239-8878-ad9595ad312a","Type":"ContainerDied","Data":"4df41a9140b8e660da049b54685a1d467e0ed205ace27ba9d47d6f689fd019ea"}
Jul 17 10:58:04.136827 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:04.136787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef28d6a-f451-4c22-adca-7179590804ea-metrics-tls\") pod \"dns-default-h4cnn\" (UID: \"bef28d6a-f451-4c22-adca-7179590804ea\") " pod="openshift-dns/dns-default-h4cnn"
Jul 17 10:58:04.136988 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:04.136940    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 10:58:04.137032 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:04.137000    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef28d6a-f451-4c22-adca-7179590804ea-metrics-tls podName:bef28d6a-f451-4c22-adca-7179590804ea nodeName:}" failed. No retries permitted until 2026-07-17 10:58:08.136985989 +0000 UTC m=+43.066525610 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bef28d6a-f451-4c22-adca-7179590804ea-metrics-tls") pod "dns-default-h4cnn" (UID: "bef28d6a-f451-4c22-adca-7179590804ea") : secret "dns-default-metrics-tls" not found
Jul 17 10:58:04.904890 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:04.904845    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:04.951497 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:04.951318    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcg7j" event={"ID":"4b3498a1-e561-4239-8878-ad9595ad312a","Type":"ContainerStarted","Data":"aafe2d87cfec8ab4060b676441c3c4bd013332f9b4bb35fbcc7a478fd16cdbfc"}
Jul 17 10:58:04.978749 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:04.978695    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pcg7j" podStartSLOduration=7.8320928389999995 podStartE2EDuration="38.978680664s" totalImagesPullingTime="31.146587825s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 10:57:26 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:58:04.977080541 +0000 UTC m=+39.906620174" watchObservedRunningTime="2026-07-17 10:58:04.978680664 +0000 UTC m=+39.908220304"
Jul 17 10:58:05.955265 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:05.955230    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gzhwh" event={"ID":"eced893c-6854-4d8c-bbc2-6d6cfa602028","Type":"ContainerStarted","Data":"233cdc8e25828935f13ecaf0730e65fd9ea9591dc4bd24cf867d50c05bb21f45"}
Jul 17 10:58:05.973938 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:05.973896    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gzhwh" podStartSLOduration=3.474083956 podStartE2EDuration="5.973881219s" totalImagesPullingTime="2.499797263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:58:05.972609831 +0000 UTC m=+40.902149473" watchObservedRunningTime="2026-07-17 10:58:05.973881219 +0000 UTC m=+40.903420838"
Jul 17 10:58:06.485513 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:06.485479    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:08.168383 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:08.168343    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef28d6a-f451-4c22-adca-7179590804ea-metrics-tls\") pod \"dns-default-h4cnn\" (UID: \"bef28d6a-f451-4c22-adca-7179590804ea\") " pod="openshift-dns/dns-default-h4cnn"
Jul 17 10:58:08.168751 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:08.168499    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 10:58:08.168751 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:08.168572    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef28d6a-f451-4c22-adca-7179590804ea-metrics-tls podName:bef28d6a-f451-4c22-adca-7179590804ea nodeName:}" failed. No retries permitted until 2026-07-17 10:58:16.168555098 +0000 UTC m=+51.098094716 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bef28d6a-f451-4c22-adca-7179590804ea-metrics-tls") pod "dns-default-h4cnn" (UID: "bef28d6a-f451-4c22-adca-7179590804ea") : secret "dns-default-metrics-tls" not found
Jul 17 10:58:16.226531 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:16.226492    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef28d6a-f451-4c22-adca-7179590804ea-metrics-tls\") pod \"dns-default-h4cnn\" (UID: \"bef28d6a-f451-4c22-adca-7179590804ea\") " pod="openshift-dns/dns-default-h4cnn"
Jul 17 10:58:16.226979 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:16.226649    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 10:58:16.226979 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:16.226727    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef28d6a-f451-4c22-adca-7179590804ea-metrics-tls podName:bef28d6a-f451-4c22-adca-7179590804ea nodeName:}" failed. No retries permitted until 2026-07-17 10:58:32.22670897 +0000 UTC m=+67.156248588 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bef28d6a-f451-4c22-adca-7179590804ea-metrics-tls") pod "dns-default-h4cnn" (UID: "bef28d6a-f451-4c22-adca-7179590804ea") : secret "dns-default-metrics-tls" not found
Jul 17 10:58:25.927686 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:25.927659    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nj5mj"
Jul 17 10:58:26.087148 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:26.087119    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:26.147770 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:26.147741    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:27.037447 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:27.037409    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:27.138141 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:27.138111    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:30.222592 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:30.222557    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9c3cb80-365e-4c10-bde0-1e9f17948a97-metrics-certs\") pod \"network-metrics-daemon-s5c9b\" (UID: \"e9c3cb80-365e-4c10-bde0-1e9f17948a97\") " pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:58:30.225231 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:30.225211    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 10:58:30.233529 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:30.233505    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 17 10:58:30.233598 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:30.233576    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9c3cb80-365e-4c10-bde0-1e9f17948a97-metrics-certs podName:e9c3cb80-365e-4c10-bde0-1e9f17948a97 nodeName:}" failed. No retries permitted until 2026-07-17 10:59:34.233553799 +0000 UTC m=+129.163093418 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9c3cb80-365e-4c10-bde0-1e9f17948a97-metrics-certs") pod "network-metrics-daemon-s5c9b" (UID: "e9c3cb80-365e-4c10-bde0-1e9f17948a97") : secret "metrics-daemon-secret" not found
Jul 17 10:58:30.322989 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:30.322958    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84ae7bd0-6126-491f-9126-2d64506af889-original-pull-secret\") pod \"global-pull-secret-syncer-b7fx4\" (UID: \"84ae7bd0-6126-491f-9126-2d64506af889\") " pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:58:30.325796 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:30.325779    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 10:58:30.337299 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:30.337263    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84ae7bd0-6126-491f-9126-2d64506af889-original-pull-secret\") pod \"global-pull-secret-syncer-b7fx4\" (UID: \"84ae7bd0-6126-491f-9126-2d64506af889\") " pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:58:30.424173 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:30.424133    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9v8m\" (UniqueName: \"kubernetes.io/projected/f2830f09-58f3-425c-a831-e5e34b542e78-kube-api-access-n9v8m\") pod \"network-check-target-g5hnv\" (UID: \"f2830f09-58f3-425c-a831-e5e34b542e78\") " pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:58:30.427325 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:30.427306    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 10:58:30.436782 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:30.436765    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 10:58:30.447399 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:30.447372    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9v8m\" (UniqueName: \"kubernetes.io/projected/f2830f09-58f3-425c-a831-e5e34b542e78-kube-api-access-n9v8m\") pod \"network-check-target-g5hnv\" (UID: \"f2830f09-58f3-425c-a831-e5e34b542e78\") " pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:58:30.476373 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:30.476268    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bvrbg\""
Jul 17 10:58:30.484174 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:30.484141    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:58:30.486430 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:30.486408    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7fx4"
Jul 17 10:58:30.625544 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:30.625513    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g5hnv"]
Jul 17 10:58:30.628908 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:58:30.628883    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2830f09_58f3_425c_a831_e5e34b542e78.slice/crio-122f0de00e5eb0f647c984732c44c0d62dbbfcb44b6791d5a440867bd2ef57e2 WatchSource:0}: Error finding container 122f0de00e5eb0f647c984732c44c0d62dbbfcb44b6791d5a440867bd2ef57e2: Status 404 returned error can't find the container with id 122f0de00e5eb0f647c984732c44c0d62dbbfcb44b6791d5a440867bd2ef57e2
Jul 17 10:58:30.640909 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:30.640884    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b7fx4"]
Jul 17 10:58:30.643988 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:58:30.643964    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ae7bd0_6126_491f_9126_2d64506af889.slice/crio-b24ae4f253d853618d138a67e9a7ff4fdbb3457cde5cebd54f3b4e847d4c1733 WatchSource:0}: Error finding container b24ae4f253d853618d138a67e9a7ff4fdbb3457cde5cebd54f3b4e847d4c1733: Status 404 returned error can't find the container with id b24ae4f253d853618d138a67e9a7ff4fdbb3457cde5cebd54f3b4e847d4c1733
Jul 17 10:58:31.011441 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:31.011402    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b7fx4" event={"ID":"84ae7bd0-6126-491f-9126-2d64506af889","Type":"ContainerStarted","Data":"b24ae4f253d853618d138a67e9a7ff4fdbb3457cde5cebd54f3b4e847d4c1733"}
Jul 17 10:58:31.012349 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:31.012329    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g5hnv" event={"ID":"f2830f09-58f3-425c-a831-e5e34b542e78","Type":"ContainerStarted","Data":"122f0de00e5eb0f647c984732c44c0d62dbbfcb44b6791d5a440867bd2ef57e2"}
Jul 17 10:58:32.239307 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:32.239253    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef28d6a-f451-4c22-adca-7179590804ea-metrics-tls\") pod \"dns-default-h4cnn\" (UID: \"bef28d6a-f451-4c22-adca-7179590804ea\") " pod="openshift-dns/dns-default-h4cnn"
Jul 17 10:58:32.239748 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:32.239431    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 10:58:32.239748 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:32.239508    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef28d6a-f451-4c22-adca-7179590804ea-metrics-tls podName:bef28d6a-f451-4c22-adca-7179590804ea nodeName:}" failed. No retries permitted until 2026-07-17 10:59:04.239487501 +0000 UTC m=+99.169027136 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bef28d6a-f451-4c22-adca-7179590804ea-metrics-tls") pod "dns-default-h4cnn" (UID: "bef28d6a-f451-4c22-adca-7179590804ea") : secret "dns-default-metrics-tls" not found
Jul 17 10:58:36.027781 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:36.027742    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b7fx4" event={"ID":"84ae7bd0-6126-491f-9126-2d64506af889","Type":"ContainerStarted","Data":"af7179d7714316c00390cb7b95df2d4630005cfc4bd533b72e356b236a14c664"}
Jul 17 10:58:36.029248 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:36.029223    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g5hnv" event={"ID":"f2830f09-58f3-425c-a831-e5e34b542e78","Type":"ContainerStarted","Data":"396d572a4db7098d4264edba76b67c1037a847ab64a8f0d3c57727554b0e825a"}
Jul 17 10:58:36.029370 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:36.029351    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:58:36.046542 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:36.046494    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-b7fx4" podStartSLOduration=65.145927568 podStartE2EDuration="1m10.046464948s" totalImagesPullingTime="4.90053738s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 10:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:58:36.045488826 +0000 UTC m=+70.975028467" watchObservedRunningTime="2026-07-17 10:58:36.046464948 +0000 UTC m=+70.976004588"
Jul 17 10:58:36.062358 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:36.062311    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g5hnv" podStartSLOduration=65.158541634 podStartE2EDuration="1m10.062298167s" totalImagesPullingTime="4.903756533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:58:36.061268044 +0000 UTC m=+70.990807683" watchObservedRunningTime="2026-07-17 10:58:36.062298167 +0000 UTC m=+70.991837808"
Jul 17 10:58:42.776143 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.776103    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5xw7l"]
Jul 17 10:58:42.777947 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.777930    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5xw7l"
Jul 17 10:58:42.786103 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.786061    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 17 10:58:42.786247 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.786086    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 17 10:58:42.786247 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.786141    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-n9jsp\""
Jul 17 10:58:42.786247 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.786164    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 17 10:58:42.786365 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.786246    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 17 10:58:42.797210 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.797186    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5xw7l"]
Jul 17 10:58:42.809689 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.809655    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87xvm\" (UniqueName: \"kubernetes.io/projected/f8f5769f-44b8-49e1-acc9-35fc5b558035-kube-api-access-87xvm\") pod \"cluster-monitoring-operator-76cf8966f8-5xw7l\" (UID: \"f8f5769f-44b8-49e1-acc9-35fc5b558035\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5xw7l"
Jul 17 10:58:42.809855 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.809764    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8f5769f-44b8-49e1-acc9-35fc5b558035-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-5xw7l\" (UID: \"f8f5769f-44b8-49e1-acc9-35fc5b558035\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5xw7l"
Jul 17 10:58:42.809855 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.809801    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f8f5769f-44b8-49e1-acc9-35fc5b558035-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-5xw7l\" (UID: \"f8f5769f-44b8-49e1-acc9-35fc5b558035\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5xw7l"
Jul 17 10:58:42.868350 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.868313    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66458c5c5-xdg6l"]
Jul 17 10:58:42.870193 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.870175    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:42.872723 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.872706    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 17 10:58:42.873106 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.873089    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nzx7f\""
Jul 17 10:58:42.874216 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.874196    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 17 10:58:42.874406 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.874393    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 17 10:58:42.878368 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.878343    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 17 10:58:42.892046 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.892016    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66458c5c5-xdg6l"]
Jul 17 10:58:42.910538 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.910504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87xvm\" (UniqueName: \"kubernetes.io/projected/f8f5769f-44b8-49e1-acc9-35fc5b558035-kube-api-access-87xvm\") pod \"cluster-monitoring-operator-76cf8966f8-5xw7l\" (UID: \"f8f5769f-44b8-49e1-acc9-35fc5b558035\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5xw7l"
Jul 17 10:58:42.910538 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.910543    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/55297f57-dfcc-4655-87ae-ca25456a8265-image-registry-private-configuration\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:42.910738 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.910581    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-bound-sa-token\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:42.910738 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.910611    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/55297f57-dfcc-4655-87ae-ca25456a8265-ca-trust-extracted\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:42.910738 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.910626    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/55297f57-dfcc-4655-87ae-ca25456a8265-trusted-ca\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:42.910738 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.910642    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/55297f57-dfcc-4655-87ae-ca25456a8265-installation-pull-secrets\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:42.910738 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.910695    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8f5769f-44b8-49e1-acc9-35fc5b558035-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-5xw7l\" (UID: \"f8f5769f-44b8-49e1-acc9-35fc5b558035\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5xw7l"
Jul 17 10:58:42.910932 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.910741    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-registry-tls\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:42.910932 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.910763    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/55297f57-dfcc-4655-87ae-ca25456a8265-registry-certificates\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:42.910932 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.910797    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f8f5769f-44b8-49e1-acc9-35fc5b558035-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-5xw7l\" (UID: \"f8f5769f-44b8-49e1-acc9-35fc5b558035\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5xw7l"
Jul 17 10:58:42.910932 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:42.910812    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 10:58:42.910932 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.910814    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qwks\" (UniqueName: \"kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-kube-api-access-6qwks\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:42.910932 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:42.910889    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8f5769f-44b8-49e1-acc9-35fc5b558035-cluster-monitoring-operator-tls podName:f8f5769f-44b8-49e1-acc9-35fc5b558035 nodeName:}" failed. No retries permitted until 2026-07-17 10:58:43.410869064 +0000 UTC m=+78.340408685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f8f5769f-44b8-49e1-acc9-35fc5b558035-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-5xw7l" (UID: "f8f5769f-44b8-49e1-acc9-35fc5b558035") : secret "cluster-monitoring-operator-tls" not found
Jul 17 10:58:42.911468 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.911452    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f8f5769f-44b8-49e1-acc9-35fc5b558035-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-5xw7l\" (UID: \"f8f5769f-44b8-49e1-acc9-35fc5b558035\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5xw7l"
Jul 17 10:58:42.923130 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.923103    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87xvm\" (UniqueName: \"kubernetes.io/projected/f8f5769f-44b8-49e1-acc9-35fc5b558035-kube-api-access-87xvm\") pod \"cluster-monitoring-operator-76cf8966f8-5xw7l\" (UID: \"f8f5769f-44b8-49e1-acc9-35fc5b558035\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5xw7l"
Jul 17 10:58:42.970198 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.970167    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-h94fh"]
Jul 17 10:58:42.972135 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.972117    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-h94fh"
Jul 17 10:58:42.977517 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.977492    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 17 10:58:42.977846 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.977816    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 17 10:58:42.977952 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.977848    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vfw7r\""
Jul 17 10:58:42.977952 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.977916    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 17 10:58:42.978041 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.977953    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 17 10:58:42.986391 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:42.986341    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-h94fh"]
Jul 17 10:58:43.011891 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.011850    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/55297f57-dfcc-4655-87ae-ca25456a8265-image-registry-private-configuration\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:43.012090 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.011931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-bound-sa-token\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:43.012090 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.011980    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7cd8f40f-2f2d-47e1-b464-33828197574b-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-h94fh\" (UID: \"7cd8f40f-2f2d-47e1-b464-33828197574b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-h94fh"
Jul 17 10:58:43.012090 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.012005    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/55297f57-dfcc-4655-87ae-ca25456a8265-ca-trust-extracted\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:43.012090 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.012025    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/55297f57-dfcc-4655-87ae-ca25456a8265-trusted-ca\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:43.012090 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.012047    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/55297f57-dfcc-4655-87ae-ca25456a8265-installation-pull-secrets\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:43.012090 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.012088    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbmx9\" (UniqueName: \"kubernetes.io/projected/7cd8f40f-2f2d-47e1-b464-33828197574b-kube-api-access-zbmx9\") pod \"kube-storage-version-migrator-operator-7658d9b749-h94fh\" (UID: \"7cd8f40f-2f2d-47e1-b464-33828197574b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-h94fh"
Jul 17 10:58:43.012393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.012149    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-registry-tls\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:43.012393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.012180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/55297f57-dfcc-4655-87ae-ca25456a8265-registry-certificates\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:43.012393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.012205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7cd8f40f-2f2d-47e1-b464-33828197574b-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-h94fh\" (UID: \"7cd8f40f-2f2d-47e1-b464-33828197574b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-h94fh"
Jul 17 10:58:43.012393 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.012232    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qwks\" (UniqueName: \"kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-kube-api-access-6qwks\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:43.012393 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:43.012351    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 10:58:43.012393 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:43.012369    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66458c5c5-xdg6l: secret "image-registry-tls" not found
Jul 17 10:58:43.012697 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:43.012426    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-registry-tls podName:55297f57-dfcc-4655-87ae-ca25456a8265 nodeName:}" failed. No retries permitted until 2026-07-17 10:58:43.512405355 +0000 UTC m=+78.441944987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-registry-tls") pod "image-registry-66458c5c5-xdg6l" (UID: "55297f57-dfcc-4655-87ae-ca25456a8265") : secret "image-registry-tls" not found
Jul 17 10:58:43.012697 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.012517    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/55297f57-dfcc-4655-87ae-ca25456a8265-ca-trust-extracted\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:43.012969 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.012944    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/55297f57-dfcc-4655-87ae-ca25456a8265-registry-certificates\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:43.013046 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.013019    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/55297f57-dfcc-4655-87ae-ca25456a8265-trusted-ca\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:43.014584 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.014557    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/55297f57-dfcc-4655-87ae-ca25456a8265-installation-pull-secrets\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:43.014699 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.014605    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/55297f57-dfcc-4655-87ae-ca25456a8265-image-registry-private-configuration\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:43.030643 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.030562    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-bound-sa-token\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:43.031053 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.031033    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qwks\" (UniqueName: \"kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-kube-api-access-6qwks\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:43.112865 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.112825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7cd8f40f-2f2d-47e1-b464-33828197574b-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-h94fh\" (UID: \"7cd8f40f-2f2d-47e1-b464-33828197574b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-h94fh"
Jul 17 10:58:43.113012 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.112910    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7cd8f40f-2f2d-47e1-b464-33828197574b-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-h94fh\" (UID: \"7cd8f40f-2f2d-47e1-b464-33828197574b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-h94fh"
Jul 17 10:58:43.113012 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.112937    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbmx9\" (UniqueName: \"kubernetes.io/projected/7cd8f40f-2f2d-47e1-b464-33828197574b-kube-api-access-zbmx9\") pod \"kube-storage-version-migrator-operator-7658d9b749-h94fh\" (UID: \"7cd8f40f-2f2d-47e1-b464-33828197574b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-h94fh"
Jul 17 10:58:43.113493 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.113467    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7cd8f40f-2f2d-47e1-b464-33828197574b-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-h94fh\" (UID: \"7cd8f40f-2f2d-47e1-b464-33828197574b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-h94fh"
Jul 17 10:58:43.115039 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.115021    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7cd8f40f-2f2d-47e1-b464-33828197574b-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-h94fh\" (UID: \"7cd8f40f-2f2d-47e1-b464-33828197574b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-h94fh"
Jul 17 10:58:43.122672 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.122640    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbmx9\" (UniqueName: \"kubernetes.io/projected/7cd8f40f-2f2d-47e1-b464-33828197574b-kube-api-access-zbmx9\") pod \"kube-storage-version-migrator-operator-7658d9b749-h94fh\" (UID: \"7cd8f40f-2f2d-47e1-b464-33828197574b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-h94fh"
Jul 17 10:58:43.185737 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.185704    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:43.281774 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.281683    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-h94fh"
Jul 17 10:58:43.286257 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.286233    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:43.404673 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.404644    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-h94fh"]
Jul 17 10:58:43.407945 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:58:43.407916    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd8f40f_2f2d_47e1_b464_33828197574b.slice/crio-3d0c8eedf7b77b1364bd969aafba35864bcec830afc3986e0c6544664620597d WatchSource:0}: Error finding container 3d0c8eedf7b77b1364bd969aafba35864bcec830afc3986e0c6544664620597d: Status 404 returned error can't find the container with id 3d0c8eedf7b77b1364bd969aafba35864bcec830afc3986e0c6544664620597d
Jul 17 10:58:43.415525 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.415495    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8f5769f-44b8-49e1-acc9-35fc5b558035-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-5xw7l\" (UID: \"f8f5769f-44b8-49e1-acc9-35fc5b558035\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5xw7l"
Jul 17 10:58:43.415654 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:43.415638    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 10:58:43.415706 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:43.415697    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8f5769f-44b8-49e1-acc9-35fc5b558035-cluster-monitoring-operator-tls podName:f8f5769f-44b8-49e1-acc9-35fc5b558035 nodeName:}" failed. No retries permitted until 2026-07-17 10:58:44.415682338 +0000 UTC m=+79.345221956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f8f5769f-44b8-49e1-acc9-35fc5b558035-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-5xw7l" (UID: "f8f5769f-44b8-49e1-acc9-35fc5b558035") : secret "cluster-monitoring-operator-tls" not found
Jul 17 10:58:43.516162 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:43.516121    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-registry-tls\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:43.516341 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:43.516271    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 10:58:43.516341 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:43.516291    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66458c5c5-xdg6l: secret "image-registry-tls" not found
Jul 17 10:58:43.516419 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:43.516345    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-registry-tls podName:55297f57-dfcc-4655-87ae-ca25456a8265 nodeName:}" failed. No retries permitted until 2026-07-17 10:58:44.516330213 +0000 UTC m=+79.445869832 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-registry-tls") pod "image-registry-66458c5c5-xdg6l" (UID: "55297f57-dfcc-4655-87ae-ca25456a8265") : secret "image-registry-tls" not found
Jul 17 10:58:44.052429 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:44.052388    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-h94fh" event={"ID":"7cd8f40f-2f2d-47e1-b464-33828197574b","Type":"ContainerStarted","Data":"3d0c8eedf7b77b1364bd969aafba35864bcec830afc3986e0c6544664620597d"}
Jul 17 10:58:44.425499 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:44.425448    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8f5769f-44b8-49e1-acc9-35fc5b558035-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-5xw7l\" (UID: \"f8f5769f-44b8-49e1-acc9-35fc5b558035\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5xw7l"
Jul 17 10:58:44.425687 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:44.425597    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 10:58:44.425687 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:44.425664    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8f5769f-44b8-49e1-acc9-35fc5b558035-cluster-monitoring-operator-tls podName:f8f5769f-44b8-49e1-acc9-35fc5b558035 nodeName:}" failed. No retries permitted until 2026-07-17 10:58:46.425647212 +0000 UTC m=+81.355186831 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f8f5769f-44b8-49e1-acc9-35fc5b558035-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-5xw7l" (UID: "f8f5769f-44b8-49e1-acc9-35fc5b558035") : secret "cluster-monitoring-operator-tls" not found
Jul 17 10:58:44.526333 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:44.526297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-registry-tls\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:44.526493 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:44.526453    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 10:58:44.526493 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:44.526473    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66458c5c5-xdg6l: secret "image-registry-tls" not found
Jul 17 10:58:44.526560 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:44.526525    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-registry-tls podName:55297f57-dfcc-4655-87ae-ca25456a8265 nodeName:}" failed. No retries permitted until 2026-07-17 10:58:46.526511051 +0000 UTC m=+81.456050671 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-registry-tls") pod "image-registry-66458c5c5-xdg6l" (UID: "55297f57-dfcc-4655-87ae-ca25456a8265") : secret "image-registry-tls" not found
Jul 17 10:58:45.176895 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:45.176862    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:45.234617 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:45.234585    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:46.057895 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:46.057862    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-h94fh" event={"ID":"7cd8f40f-2f2d-47e1-b464-33828197574b","Type":"ContainerStarted","Data":"3bc19463bc32b4fc02b4cd51690290469753d9ce5a6724d67dc9a43ec153d847"}
Jul 17 10:58:46.092916 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:46.092874    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-h94fh" podStartSLOduration=2.459467889 podStartE2EDuration="4.092860601s" totalImagesPullingTime="1.633392712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:58:46.090543644 +0000 UTC m=+81.020083285" watchObservedRunningTime="2026-07-17 10:58:46.092860601 +0000 UTC m=+81.022400241"
Jul 17 10:58:46.225308 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:46.225279    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:46.318443 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:46.318362    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:46.393569 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:46.393540    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:46.442408 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:46.442375    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8f5769f-44b8-49e1-acc9-35fc5b558035-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-5xw7l\" (UID: \"f8f5769f-44b8-49e1-acc9-35fc5b558035\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5xw7l"
Jul 17 10:58:46.442565 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:46.442523    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 10:58:46.442604 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:46.442585    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8f5769f-44b8-49e1-acc9-35fc5b558035-cluster-monitoring-operator-tls podName:f8f5769f-44b8-49e1-acc9-35fc5b558035 nodeName:}" failed. No retries permitted until 2026-07-17 10:58:50.442568783 +0000 UTC m=+85.372108409 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f8f5769f-44b8-49e1-acc9-35fc5b558035-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-5xw7l" (UID: "f8f5769f-44b8-49e1-acc9-35fc5b558035") : secret "cluster-monitoring-operator-tls" not found
Jul 17 10:58:46.543004 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:46.542965    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-registry-tls\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:46.543183 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:46.543137    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 10:58:46.543183 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:46.543155    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66458c5c5-xdg6l: secret "image-registry-tls" not found
Jul 17 10:58:46.543255 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:46.543209    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-registry-tls podName:55297f57-dfcc-4655-87ae-ca25456a8265 nodeName:}" failed. No retries permitted until 2026-07-17 10:58:50.54319188 +0000 UTC m=+85.472731517 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-registry-tls") pod "image-registry-66458c5c5-xdg6l" (UID: "55297f57-dfcc-4655-87ae-ca25456a8265") : secret "image-registry-tls" not found
Jul 17 10:58:46.816902 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:46.816869    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:46.917683 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:46.917652    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:47.098778 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:47.098703    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gf598"]
Jul 17 10:58:47.100756 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:47.100740    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gf598"
Jul 17 10:58:47.106577 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:47.106296    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 17 10:58:47.106577 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:47.106376    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 17 10:58:47.106577 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:47.106398    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gfnlk\""
Jul 17 10:58:47.106577 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:47.106531    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 17 10:58:47.106764 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:47.106605    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 17 10:58:47.115780 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:47.115757    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gf598"]
Jul 17 10:58:47.128580 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:47.128555    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:47.148174 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:47.148147    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf372042-c2bc-45b7-a95d-8a90116a13a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gf598\" (UID: \"bf372042-c2bc-45b7-a95d-8a90116a13a7\") " pod="openshift-insights/insights-runtime-extractor-gf598"
Jul 17 10:58:47.148285 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:47.148268    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf372042-c2bc-45b7-a95d-8a90116a13a7-crio-socket\") pod \"insights-runtime-extractor-gf598\" (UID: \"bf372042-c2bc-45b7-a95d-8a90116a13a7\") " pod="openshift-insights/insights-runtime-extractor-gf598"
Jul 17 10:58:47.148331 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:47.148311    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtrz9\" (UniqueName: \"kubernetes.io/projected/bf372042-c2bc-45b7-a95d-8a90116a13a7-kube-api-access-vtrz9\") pod \"insights-runtime-extractor-gf598\" (UID: \"bf372042-c2bc-45b7-a95d-8a90116a13a7\") " pod="openshift-insights/insights-runtime-extractor-gf598"
Jul 17 10:58:47.148363 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:47.148331    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf372042-c2bc-45b7-a95d-8a90116a13a7-data-volume\") pod \"insights-runtime-extractor-gf598\" (UID: \"bf372042-c2bc-45b7-a95d-8a90116a13a7\") " pod="openshift-insights/insights-runtime-extractor-gf598"
Jul 17 10:58:47.148363 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:47.148350    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf372042-c2bc-45b7-a95d-8a90116a13a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gf598\" (UID: \"bf372042-c2bc-45b7-a95d-8a90116a13a7\") " pod="openshift-insights/insights-runtime-extractor-gf598"
Jul 17 10:58:47.249238 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:47.249195    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf372042-c2bc-45b7-a95d-8a90116a13a7-crio-socket\") pod \"insights-runtime-extractor-gf598\" (UID: \"bf372042-c2bc-45b7-a95d-8a90116a13a7\") " pod="openshift-insights/insights-runtime-extractor-gf598"
Jul 17 10:58:47.249238 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:47.249245    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtrz9\" (UniqueName: \"kubernetes.io/projected/bf372042-c2bc-45b7-a95d-8a90116a13a7-kube-api-access-vtrz9\") pod \"insights-runtime-extractor-gf598\" (UID: \"bf372042-c2bc-45b7-a95d-8a90116a13a7\") " pod="openshift-insights/insights-runtime-extractor-gf598"
Jul 17 10:58:47.249624 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:47.249264    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf372042-c2bc-45b7-a95d-8a90116a13a7-data-volume\") pod \"insights-runtime-extractor-gf598\" (UID: \"bf372042-c2bc-45b7-a95d-8a90116a13a7\") " pod="openshift-insights/insights-runtime-extractor-gf598"
Jul 17 10:58:47.249624 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:47.249288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf372042-c2bc-45b7-a95d-8a90116a13a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gf598\" (UID: \"bf372042-c2bc-45b7-a95d-8a90116a13a7\") " pod="openshift-insights/insights-runtime-extractor-gf598"
Jul 17 10:58:47.249624 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:47.249328    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf372042-c2bc-45b7-a95d-8a90116a13a7-crio-socket\") pod \"insights-runtime-extractor-gf598\" (UID: \"bf372042-c2bc-45b7-a95d-8a90116a13a7\") " pod="openshift-insights/insights-runtime-extractor-gf598"
Jul 17 10:58:47.249624 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:47.249346    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf372042-c2bc-45b7-a95d-8a90116a13a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gf598\" (UID: \"bf372042-c2bc-45b7-a95d-8a90116a13a7\") " pod="openshift-insights/insights-runtime-extractor-gf598"
Jul 17 10:58:47.249624 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:47.249416    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 10:58:47.249624 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:47.249426    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 10:58:47.249624 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:47.249475    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf372042-c2bc-45b7-a95d-8a90116a13a7-insights-runtime-extractor-tls podName:bf372042-c2bc-45b7-a95d-8a90116a13a7 nodeName:}" failed. No retries permitted until 2026-07-17 10:58:47.749457254 +0000 UTC m=+82.678996873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf372042-c2bc-45b7-a95d-8a90116a13a7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gf598" (UID: "bf372042-c2bc-45b7-a95d-8a90116a13a7") : secret "insights-runtime-extractor-tls" not found
Jul 17 10:58:47.249624 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:47.249501    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf372042-c2bc-45b7-a95d-8a90116a13a7-kube-rbac-proxy-cm podName:bf372042-c2bc-45b7-a95d-8a90116a13a7 nodeName:}" failed. No retries permitted until 2026-07-17 10:58:47.749490687 +0000 UTC m=+82.679030322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf372042-c2bc-45b7-a95d-8a90116a13a7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gf598" (UID: "bf372042-c2bc-45b7-a95d-8a90116a13a7") : configmap "kube-rbac-proxy" not found
Jul 17 10:58:47.249624 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:47.249592    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf372042-c2bc-45b7-a95d-8a90116a13a7-data-volume\") pod \"insights-runtime-extractor-gf598\" (UID: \"bf372042-c2bc-45b7-a95d-8a90116a13a7\") " pod="openshift-insights/insights-runtime-extractor-gf598"
Jul 17 10:58:47.259667 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:47.259641    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtrz9\" (UniqueName: \"kubernetes.io/projected/bf372042-c2bc-45b7-a95d-8a90116a13a7-kube-api-access-vtrz9\") pod \"insights-runtime-extractor-gf598\" (UID: \"bf372042-c2bc-45b7-a95d-8a90116a13a7\") " pod="openshift-insights/insights-runtime-extractor-gf598"
Jul 17 10:58:47.547229 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:47.547193    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:47.616210 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:47.616176    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:47.752876 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:47.752841    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf372042-c2bc-45b7-a95d-8a90116a13a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gf598\" (UID: \"bf372042-c2bc-45b7-a95d-8a90116a13a7\") " pod="openshift-insights/insights-runtime-extractor-gf598"
Jul 17 10:58:47.753058 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:47.752897    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf372042-c2bc-45b7-a95d-8a90116a13a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gf598\" (UID: \"bf372042-c2bc-45b7-a95d-8a90116a13a7\") " pod="openshift-insights/insights-runtime-extractor-gf598"
Jul 17 10:58:47.753058 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:47.752954    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 10:58:47.753058 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:47.753013    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf372042-c2bc-45b7-a95d-8a90116a13a7-kube-rbac-proxy-cm podName:bf372042-c2bc-45b7-a95d-8a90116a13a7 nodeName:}" failed. No retries permitted until 2026-07-17 10:58:48.752998708 +0000 UTC m=+83.682538330 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf372042-c2bc-45b7-a95d-8a90116a13a7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gf598" (UID: "bf372042-c2bc-45b7-a95d-8a90116a13a7") : configmap "kube-rbac-proxy" not found
Jul 17 10:58:47.753058 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:47.753039    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 10:58:47.753255 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:47.753117    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf372042-c2bc-45b7-a95d-8a90116a13a7-insights-runtime-extractor-tls podName:bf372042-c2bc-45b7-a95d-8a90116a13a7 nodeName:}" failed. No retries permitted until 2026-07-17 10:58:48.753103859 +0000 UTC m=+83.682643499 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf372042-c2bc-45b7-a95d-8a90116a13a7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gf598" (UID: "bf372042-c2bc-45b7-a95d-8a90116a13a7") : secret "insights-runtime-extractor-tls" not found
Jul 17 10:58:48.200709 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:48.200673    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:48.262121 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:48.262090    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:48.761756 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:48.761710    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf372042-c2bc-45b7-a95d-8a90116a13a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gf598\" (UID: \"bf372042-c2bc-45b7-a95d-8a90116a13a7\") " pod="openshift-insights/insights-runtime-extractor-gf598"
Jul 17 10:58:48.761930 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:48.761781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf372042-c2bc-45b7-a95d-8a90116a13a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gf598\" (UID: \"bf372042-c2bc-45b7-a95d-8a90116a13a7\") " pod="openshift-insights/insights-runtime-extractor-gf598"
Jul 17 10:58:48.761930 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:48.761843    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 10:58:48.761930 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:48.761874    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 10:58:48.762131 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:48.761935    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf372042-c2bc-45b7-a95d-8a90116a13a7-kube-rbac-proxy-cm podName:bf372042-c2bc-45b7-a95d-8a90116a13a7 nodeName:}" failed. No retries permitted until 2026-07-17 10:58:50.76190752 +0000 UTC m=+85.691447140 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf372042-c2bc-45b7-a95d-8a90116a13a7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gf598" (UID: "bf372042-c2bc-45b7-a95d-8a90116a13a7") : configmap "kube-rbac-proxy" not found
Jul 17 10:58:48.762131 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:48.761971    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf372042-c2bc-45b7-a95d-8a90116a13a7-insights-runtime-extractor-tls podName:bf372042-c2bc-45b7-a95d-8a90116a13a7 nodeName:}" failed. No retries permitted until 2026-07-17 10:58:50.761946016 +0000 UTC m=+85.691485635 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf372042-c2bc-45b7-a95d-8a90116a13a7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gf598" (UID: "bf372042-c2bc-45b7-a95d-8a90116a13a7") : secret "insights-runtime-extractor-tls" not found
Jul 17 10:58:49.129455 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:49.129383    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-s4ldw"]
Jul 17 10:58:49.132157 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:49.132141    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-s4ldw"
Jul 17 10:58:49.134680 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:49.134654    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 17 10:58:49.134790 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:49.134682    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 17 10:58:49.135677 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:49.135660    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 17 10:58:49.135677 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:49.135673    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 17 10:58:49.135959 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:49.135662    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xbl5z\""
Jul 17 10:58:49.140022 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:49.140003    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-s4ldw"]
Jul 17 10:58:49.159339 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:49.159278    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:49.165194 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:49.165176    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn66w\" (UniqueName: \"kubernetes.io/projected/1e668ecc-6d45-4c01-91e1-2cf638a58bad-kube-api-access-zn66w\") pod \"service-ca-58598695f7-s4ldw\" (UID: \"1e668ecc-6d45-4c01-91e1-2cf638a58bad\") " pod="openshift-service-ca/service-ca-58598695f7-s4ldw"
Jul 17 10:58:49.165275 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:49.165224    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1e668ecc-6d45-4c01-91e1-2cf638a58bad-signing-key\") pod \"service-ca-58598695f7-s4ldw\" (UID: \"1e668ecc-6d45-4c01-91e1-2cf638a58bad\") " pod="openshift-service-ca/service-ca-58598695f7-s4ldw"
Jul 17 10:58:49.165363 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:49.165348    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1e668ecc-6d45-4c01-91e1-2cf638a58bad-signing-cabundle\") pod \"service-ca-58598695f7-s4ldw\" (UID: \"1e668ecc-6d45-4c01-91e1-2cf638a58bad\") " pod="openshift-service-ca/service-ca-58598695f7-s4ldw"
Jul 17 10:58:49.221246 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:49.221205    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:49.266032 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:49.266008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1e668ecc-6d45-4c01-91e1-2cf638a58bad-signing-cabundle\") pod \"service-ca-58598695f7-s4ldw\" (UID: \"1e668ecc-6d45-4c01-91e1-2cf638a58bad\") " pod="openshift-service-ca/service-ca-58598695f7-s4ldw"
Jul 17 10:58:49.266390 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:49.266044    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn66w\" (UniqueName: \"kubernetes.io/projected/1e668ecc-6d45-4c01-91e1-2cf638a58bad-kube-api-access-zn66w\") pod \"service-ca-58598695f7-s4ldw\" (UID: \"1e668ecc-6d45-4c01-91e1-2cf638a58bad\") " pod="openshift-service-ca/service-ca-58598695f7-s4ldw"
Jul 17 10:58:49.266390 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:49.266141    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1e668ecc-6d45-4c01-91e1-2cf638a58bad-signing-key\") pod \"service-ca-58598695f7-s4ldw\" (UID: \"1e668ecc-6d45-4c01-91e1-2cf638a58bad\") " pod="openshift-service-ca/service-ca-58598695f7-s4ldw"
Jul 17 10:58:49.266770 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:49.266753    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1e668ecc-6d45-4c01-91e1-2cf638a58bad-signing-cabundle\") pod \"service-ca-58598695f7-s4ldw\" (UID: \"1e668ecc-6d45-4c01-91e1-2cf638a58bad\") " pod="openshift-service-ca/service-ca-58598695f7-s4ldw"
Jul 17 10:58:49.268593 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:49.268574    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1e668ecc-6d45-4c01-91e1-2cf638a58bad-signing-key\") pod \"service-ca-58598695f7-s4ldw\" (UID: \"1e668ecc-6d45-4c01-91e1-2cf638a58bad\") " pod="openshift-service-ca/service-ca-58598695f7-s4ldw"
Jul 17 10:58:49.277423 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:49.277404    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn66w\" (UniqueName: \"kubernetes.io/projected/1e668ecc-6d45-4c01-91e1-2cf638a58bad-kube-api-access-zn66w\") pod \"service-ca-58598695f7-s4ldw\" (UID: \"1e668ecc-6d45-4c01-91e1-2cf638a58bad\") " pod="openshift-service-ca/service-ca-58598695f7-s4ldw"
Jul 17 10:58:49.441701 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:49.441671    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-s4ldw"
Jul 17 10:58:49.564163 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:49.564141    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-s4ldw"]
Jul 17 10:58:49.566549 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:58:49.566522    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e668ecc_6d45_4c01_91e1_2cf638a58bad.slice/crio-32a013f97aff641aff0305ee558049806b3f4043781643909e310ec27bebd536 WatchSource:0}: Error finding container 32a013f97aff641aff0305ee558049806b3f4043781643909e310ec27bebd536: Status 404 returned error can't find the container with id 32a013f97aff641aff0305ee558049806b3f4043781643909e310ec27bebd536
Jul 17 10:58:49.892741 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:49.892720    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-87r57_f1f509b8-7e29-445b-ad25-dd54a1c6dc07/dns-node-resolver/0.log"
Jul 17 10:58:50.073499 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:50.073468    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-s4ldw" event={"ID":"1e668ecc-6d45-4c01-91e1-2cf638a58bad","Type":"ContainerStarted","Data":"32a013f97aff641aff0305ee558049806b3f4043781643909e310ec27bebd536"}
Jul 17 10:58:50.154894 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:50.154809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:50.476115 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:50.476008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8f5769f-44b8-49e1-acc9-35fc5b558035-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-5xw7l\" (UID: \"f8f5769f-44b8-49e1-acc9-35fc5b558035\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5xw7l"
Jul 17 10:58:50.476559 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:50.476187    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 10:58:50.476559 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:50.476276    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8f5769f-44b8-49e1-acc9-35fc5b558035-cluster-monitoring-operator-tls podName:f8f5769f-44b8-49e1-acc9-35fc5b558035 nodeName:}" failed. No retries permitted until 2026-07-17 10:58:58.476251199 +0000 UTC m=+93.405790823 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f8f5769f-44b8-49e1-acc9-35fc5b558035-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-5xw7l" (UID: "f8f5769f-44b8-49e1-acc9-35fc5b558035") : secret "cluster-monitoring-operator-tls" not found
Jul 17 10:58:50.577277 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:50.577245    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-registry-tls\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:50.577453 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:50.577399    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 10:58:50.577453 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:50.577418    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66458c5c5-xdg6l: secret "image-registry-tls" not found
Jul 17 10:58:50.577534 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:50.577469    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-registry-tls podName:55297f57-dfcc-4655-87ae-ca25456a8265 nodeName:}" failed. No retries permitted until 2026-07-17 10:58:58.577454805 +0000 UTC m=+93.506994427 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-registry-tls") pod "image-registry-66458c5c5-xdg6l" (UID: "55297f57-dfcc-4655-87ae-ca25456a8265") : secret "image-registry-tls" not found
Jul 17 10:58:50.778728 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:50.778643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf372042-c2bc-45b7-a95d-8a90116a13a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gf598\" (UID: \"bf372042-c2bc-45b7-a95d-8a90116a13a7\") " pod="openshift-insights/insights-runtime-extractor-gf598"
Jul 17 10:58:50.778728 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:50.778701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf372042-c2bc-45b7-a95d-8a90116a13a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gf598\" (UID: \"bf372042-c2bc-45b7-a95d-8a90116a13a7\") " pod="openshift-insights/insights-runtime-extractor-gf598"
Jul 17 10:58:50.778954 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:50.778782    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 10:58:50.778954 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:50.778851    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 10:58:50.778954 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:50.778859    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf372042-c2bc-45b7-a95d-8a90116a13a7-kube-rbac-proxy-cm podName:bf372042-c2bc-45b7-a95d-8a90116a13a7 nodeName:}" failed. No retries permitted until 2026-07-17 10:58:54.778840762 +0000 UTC m=+89.708380395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf372042-c2bc-45b7-a95d-8a90116a13a7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gf598" (UID: "bf372042-c2bc-45b7-a95d-8a90116a13a7") : configmap "kube-rbac-proxy" not found
Jul 17 10:58:50.778954 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:50.778936    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf372042-c2bc-45b7-a95d-8a90116a13a7-insights-runtime-extractor-tls podName:bf372042-c2bc-45b7-a95d-8a90116a13a7 nodeName:}" failed. No retries permitted until 2026-07-17 10:58:54.778909181 +0000 UTC m=+89.708448806 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf372042-c2bc-45b7-a95d-8a90116a13a7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gf598" (UID: "bf372042-c2bc-45b7-a95d-8a90116a13a7") : secret "insights-runtime-extractor-tls" not found
Jul 17 10:58:51.094839 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:51.094761    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qqqf9_72f5631e-c70d-47b8-9405-d8b986fd12e7/node-ca/0.log"
Jul 17 10:58:51.892084 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:51.892011    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gzhwh_eced893c-6854-4d8c-bbc2-6d6cfa602028/serve-healthcheck-canary/0.log"
Jul 17 10:58:52.047492 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:52.047454    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:52.081266 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:52.081232    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-s4ldw" event={"ID":"1e668ecc-6d45-4c01-91e1-2cf638a58bad","Type":"ContainerStarted","Data":"d6a2ec672c9f5bf48d7260a287f4b599feccb767da0d3b91121611c71918b713"}
Jul 17 10:58:52.258853 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:52.258766    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:52.359631 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:52.359563    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:52.421060 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:52.421026    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:52.498580 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:52.498543    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-h94fh_7cd8f40f-2f2d-47e1-b464-33828197574b/kube-storage-version-migrator-operator/0.log"
Jul 17 10:58:52.750484 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:52.750433    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:52.996112 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:52.996061    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:53.749681 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:53.749651    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:54.149517 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:54.149474    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:54.348548 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:54.348512    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:54.575285 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:54.575203    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:54.811422 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:54.811384    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf372042-c2bc-45b7-a95d-8a90116a13a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gf598\" (UID: \"bf372042-c2bc-45b7-a95d-8a90116a13a7\") " pod="openshift-insights/insights-runtime-extractor-gf598"
Jul 17 10:58:54.811585 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:54.811440    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf372042-c2bc-45b7-a95d-8a90116a13a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gf598\" (UID: \"bf372042-c2bc-45b7-a95d-8a90116a13a7\") " pod="openshift-insights/insights-runtime-extractor-gf598"
Jul 17 10:58:54.811585 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:54.811511    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 10:58:54.811705 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:54.811591    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 10:58:54.811705 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:54.811592    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf372042-c2bc-45b7-a95d-8a90116a13a7-kube-rbac-proxy-cm podName:bf372042-c2bc-45b7-a95d-8a90116a13a7 nodeName:}" failed. No retries permitted until 2026-07-17 10:59:02.811574983 +0000 UTC m=+97.741114602 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf372042-c2bc-45b7-a95d-8a90116a13a7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gf598" (UID: "bf372042-c2bc-45b7-a95d-8a90116a13a7") : configmap "kube-rbac-proxy" not found
Jul 17 10:58:54.811705 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:54.811688    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf372042-c2bc-45b7-a95d-8a90116a13a7-insights-runtime-extractor-tls podName:bf372042-c2bc-45b7-a95d-8a90116a13a7 nodeName:}" failed. No retries permitted until 2026-07-17 10:59:02.811674279 +0000 UTC m=+97.741213897 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf372042-c2bc-45b7-a95d-8a90116a13a7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gf598" (UID: "bf372042-c2bc-45b7-a95d-8a90116a13a7") : secret "insights-runtime-extractor-tls" not found
Jul 17 10:58:54.947874 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:54.947844    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:55.355833 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:55.355751    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:55.560557 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:55.560528    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:55.757534 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:55.757501    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:56.149584 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:56.149550    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:56.341650 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:56.341615    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:56.411535 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:56.411453    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:56.548231 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:56.548201    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:56.750764 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:56.750680    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:57.949767 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:57.949736    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:58.008544 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:58.008515    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:58.143903 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:58.143839    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:58.352244 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:58.352155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:58.546045 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:58.546010    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8f5769f-44b8-49e1-acc9-35fc5b558035-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-5xw7l\" (UID: \"f8f5769f-44b8-49e1-acc9-35fc5b558035\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5xw7l"
Jul 17 10:58:58.546251 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:58.546188    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 10:58:58.546313 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:58:58.546261    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8f5769f-44b8-49e1-acc9-35fc5b558035-cluster-monitoring-operator-tls podName:f8f5769f-44b8-49e1-acc9-35fc5b558035 nodeName:}" failed. No retries permitted until 2026-07-17 10:59:14.5462435 +0000 UTC m=+109.475783122 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f8f5769f-44b8-49e1-acc9-35fc5b558035-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-5xw7l" (UID: "f8f5769f-44b8-49e1-acc9-35fc5b558035") : secret "cluster-monitoring-operator-tls" not found
Jul 17 10:58:58.547498 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:58.547477    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:58.647715 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:58.647676    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-registry-tls\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:58.650124 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:58.650103    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-registry-tls\") pod \"image-registry-66458c5c5-xdg6l\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") " pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:58.756768 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:58.756729    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:58.779724 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:58.779694    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:58.920364 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:58.920119    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-s4ldw" podStartSLOduration=8.304575824 podStartE2EDuration="9.920097268s" totalImagesPullingTime="1.615521444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:58:52.099564649 +0000 UTC m=+87.029104286" watchObservedRunningTime="2026-07-17 10:58:58.920097268 +0000 UTC m=+93.849636912"
Jul 17 10:58:58.920598 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:58.920534    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66458c5c5-xdg6l"]
Jul 17 10:58:58.926033 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:58:58.925986    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55297f57_dfcc_4655_87ae_ca25456a8265.slice/crio-38f05e1dfe4afb602f1df8cad57efea82a9e87cf987ce91931d67058e2c6fe2c WatchSource:0}: Error finding container 38f05e1dfe4afb602f1df8cad57efea82a9e87cf987ce91931d67058e2c6fe2c: Status 404 returned error can't find the container with id 38f05e1dfe4afb602f1df8cad57efea82a9e87cf987ce91931d67058e2c6fe2c
Jul 17 10:58:59.108116 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:59.108060    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66458c5c5-xdg6l" event={"ID":"55297f57-dfcc-4655-87ae-ca25456a8265","Type":"ContainerStarted","Data":"ff7cc561a615628fe7cf2112d8f0b9191ebea323c863c4369e6d0f77ae101225"}
Jul 17 10:58:59.108590 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:59.108163    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66458c5c5-xdg6l" event={"ID":"55297f57-dfcc-4655-87ae-ca25456a8265","Type":"ContainerStarted","Data":"38f05e1dfe4afb602f1df8cad57efea82a9e87cf987ce91931d67058e2c6fe2c"}
Jul 17 10:58:59.108590 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:59.108192    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:58:59.129466 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:59.129064    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-66458c5c5-xdg6l" podStartSLOduration=17.12904879 podStartE2EDuration="17.12904879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:58:59.128523877 +0000 UTC m=+94.058063517" watchObservedRunningTime="2026-07-17 10:58:59.12904879 +0000 UTC m=+94.058588431"
Jul 17 10:58:59.344135 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:59.344027    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:59.471779 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:59.471749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:58:59.576964 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:58:59.576924    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:00.349183 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:00.349135    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:00.948289 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:00.948245    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:01.146574 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:01.146539    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:01.347860 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:01.347774    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:01.565185 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:01.565156    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:01.757937 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:01.757906    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:02.359870 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:02.359836    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:02.548267 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:02.548219    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:02.815781 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:02.815748    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:02.887459 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:02.887430    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf372042-c2bc-45b7-a95d-8a90116a13a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gf598\" (UID: \"bf372042-c2bc-45b7-a95d-8a90116a13a7\") " pod="openshift-insights/insights-runtime-extractor-gf598"
Jul 17 10:59:02.887585 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:02.887486    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf372042-c2bc-45b7-a95d-8a90116a13a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gf598\" (UID: \"bf372042-c2bc-45b7-a95d-8a90116a13a7\") " pod="openshift-insights/insights-runtime-extractor-gf598"
Jul 17 10:59:02.888187 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:02.888170    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf372042-c2bc-45b7-a95d-8a90116a13a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gf598\" (UID: \"bf372042-c2bc-45b7-a95d-8a90116a13a7\") " pod="openshift-insights/insights-runtime-extractor-gf598"
Jul 17 10:59:02.889815 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:02.889798    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf372042-c2bc-45b7-a95d-8a90116a13a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gf598\" (UID: \"bf372042-c2bc-45b7-a95d-8a90116a13a7\") " pod="openshift-insights/insights-runtime-extractor-gf598"
Jul 17 10:59:03.008756 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:03.008718    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gf598"
Jul 17 10:59:03.145457 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:03.145423    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gf598"]
Jul 17 10:59:03.148857 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:59:03.148826    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf372042_c2bc_45b7_a95d_8a90116a13a7.slice/crio-1ccb73169db95c251220158c6477ffa046d3b2d3a0f5fea894c7fd53aad80ec8 WatchSource:0}: Error finding container 1ccb73169db95c251220158c6477ffa046d3b2d3a0f5fea894c7fd53aad80ec8: Status 404 returned error can't find the container with id 1ccb73169db95c251220158c6477ffa046d3b2d3a0f5fea894c7fd53aad80ec8
Jul 17 10:59:04.130753 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:04.130712    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gf598" event={"ID":"bf372042-c2bc-45b7-a95d-8a90116a13a7","Type":"ContainerStarted","Data":"1ccb73169db95c251220158c6477ffa046d3b2d3a0f5fea894c7fd53aad80ec8"}
Jul 17 10:59:04.299786 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:04.299742    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef28d6a-f451-4c22-adca-7179590804ea-metrics-tls\") pod \"dns-default-h4cnn\" (UID: \"bef28d6a-f451-4c22-adca-7179590804ea\") " pod="openshift-dns/dns-default-h4cnn"
Jul 17 10:59:04.302336 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:04.302307    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef28d6a-f451-4c22-adca-7179590804ea-metrics-tls\") pod \"dns-default-h4cnn\" (UID: \"bef28d6a-f451-4c22-adca-7179590804ea\") " pod="openshift-dns/dns-default-h4cnn"
Jul 17 10:59:04.345288 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:04.345256    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w544b\""
Jul 17 10:59:04.353416 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:04.353373    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h4cnn"
Jul 17 10:59:04.500647 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:04.500524    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h4cnn"]
Jul 17 10:59:04.503041 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:59:04.503010    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef28d6a_f451_4c22_adca_7179590804ea.slice/crio-2654c570b388d0a01175bc2551a71ced80d5026a55850a54013117da5fab0996 WatchSource:0}: Error finding container 2654c570b388d0a01175bc2551a71ced80d5026a55850a54013117da5fab0996: Status 404 returned error can't find the container with id 2654c570b388d0a01175bc2551a71ced80d5026a55850a54013117da5fab0996
Jul 17 10:59:05.135701 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:05.135663    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h4cnn" event={"ID":"bef28d6a-f451-4c22-adca-7179590804ea","Type":"ContainerStarted","Data":"2654c570b388d0a01175bc2551a71ced80d5026a55850a54013117da5fab0996"}
Jul 17 10:59:05.998866 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:05.998827    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:06.058937 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:06.058900    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:07.035654 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:07.035624    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g5hnv"
Jul 17 10:59:08.834251 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:08.834208    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:08.928465 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:08.928440    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:09.110770 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:09.110680    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:09.150193 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:09.150161    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h4cnn" event={"ID":"bef28d6a-f451-4c22-adca-7179590804ea","Type":"ContainerStarted","Data":"252e43fda92bd245d085749d1635055263d1637796b51d53eed419c90029e91f"}
Jul 17 10:59:09.273768 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:09.273725    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:09.374032 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:09.373944    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:09.492005 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:09.491940    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:09.596258 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:09.596227    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:09.777728 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:09.777692    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66458c5c5-xdg6l"]
Jul 17 10:59:09.794438 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:09.794406    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:09.822702 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:09.822664    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7fc569f596-rgpxb"]
Jul 17 10:59:09.838976 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:09.838945    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:09.845836 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:09.845811    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fc569f596-rgpxb"]
Jul 17 10:59:09.899803 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:09.899767    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:09.947255 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:09.947202    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj26q\" (UniqueName: \"kubernetes.io/projected/632083f5-6ec0-4455-8b41-f372f69294fc-kube-api-access-pj26q\") pod \"image-registry-7fc569f596-rgpxb\" (UID: \"632083f5-6ec0-4455-8b41-f372f69294fc\") " pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:09.947432 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:09.947268    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/632083f5-6ec0-4455-8b41-f372f69294fc-image-registry-private-configuration\") pod \"image-registry-7fc569f596-rgpxb\" (UID: \"632083f5-6ec0-4455-8b41-f372f69294fc\") " pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:09.947432 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:09.947302    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/632083f5-6ec0-4455-8b41-f372f69294fc-registry-certificates\") pod \"image-registry-7fc569f596-rgpxb\" (UID: \"632083f5-6ec0-4455-8b41-f372f69294fc\") " pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:09.947432 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:09.947327    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/632083f5-6ec0-4455-8b41-f372f69294fc-trusted-ca\") pod \"image-registry-7fc569f596-rgpxb\" (UID: \"632083f5-6ec0-4455-8b41-f372f69294fc\") " pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:09.947432 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:09.947371    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/632083f5-6ec0-4455-8b41-f372f69294fc-bound-sa-token\") pod \"image-registry-7fc569f596-rgpxb\" (UID: \"632083f5-6ec0-4455-8b41-f372f69294fc\") " pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:09.947432 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:09.947403    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/632083f5-6ec0-4455-8b41-f372f69294fc-installation-pull-secrets\") pod \"image-registry-7fc569f596-rgpxb\" (UID: \"632083f5-6ec0-4455-8b41-f372f69294fc\") " pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:09.947629 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:09.947451    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/632083f5-6ec0-4455-8b41-f372f69294fc-registry-tls\") pod \"image-registry-7fc569f596-rgpxb\" (UID: \"632083f5-6ec0-4455-8b41-f372f69294fc\") " pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:09.947629 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:09.947486    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/632083f5-6ec0-4455-8b41-f372f69294fc-ca-trust-extracted\") pod \"image-registry-7fc569f596-rgpxb\" (UID: \"632083f5-6ec0-4455-8b41-f372f69294fc\") " pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:10.001536 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.001500    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:10.048566 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.048471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj26q\" (UniqueName: \"kubernetes.io/projected/632083f5-6ec0-4455-8b41-f372f69294fc-kube-api-access-pj26q\") pod \"image-registry-7fc569f596-rgpxb\" (UID: \"632083f5-6ec0-4455-8b41-f372f69294fc\") " pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:10.048566 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.048523    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/632083f5-6ec0-4455-8b41-f372f69294fc-image-registry-private-configuration\") pod \"image-registry-7fc569f596-rgpxb\" (UID: \"632083f5-6ec0-4455-8b41-f372f69294fc\") " pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:10.048794 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.048570    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/632083f5-6ec0-4455-8b41-f372f69294fc-registry-certificates\") pod \"image-registry-7fc569f596-rgpxb\" (UID: \"632083f5-6ec0-4455-8b41-f372f69294fc\") " pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:10.048794 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.048588    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/632083f5-6ec0-4455-8b41-f372f69294fc-trusted-ca\") pod \"image-registry-7fc569f596-rgpxb\" (UID: \"632083f5-6ec0-4455-8b41-f372f69294fc\") " pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:10.048794 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.048626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/632083f5-6ec0-4455-8b41-f372f69294fc-bound-sa-token\") pod \"image-registry-7fc569f596-rgpxb\" (UID: \"632083f5-6ec0-4455-8b41-f372f69294fc\") " pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:10.048794 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.048650    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/632083f5-6ec0-4455-8b41-f372f69294fc-installation-pull-secrets\") pod \"image-registry-7fc569f596-rgpxb\" (UID: \"632083f5-6ec0-4455-8b41-f372f69294fc\") " pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:10.048794 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.048703    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/632083f5-6ec0-4455-8b41-f372f69294fc-registry-tls\") pod \"image-registry-7fc569f596-rgpxb\" (UID: \"632083f5-6ec0-4455-8b41-f372f69294fc\") " pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:10.048794 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.048729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/632083f5-6ec0-4455-8b41-f372f69294fc-ca-trust-extracted\") pod \"image-registry-7fc569f596-rgpxb\" (UID: \"632083f5-6ec0-4455-8b41-f372f69294fc\") " pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:10.049138 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.049117    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/632083f5-6ec0-4455-8b41-f372f69294fc-ca-trust-extracted\") pod \"image-registry-7fc569f596-rgpxb\" (UID: \"632083f5-6ec0-4455-8b41-f372f69294fc\") " pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:10.049665 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.049636    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/632083f5-6ec0-4455-8b41-f372f69294fc-registry-certificates\") pod \"image-registry-7fc569f596-rgpxb\" (UID: \"632083f5-6ec0-4455-8b41-f372f69294fc\") " pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:10.049763 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.049649    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/632083f5-6ec0-4455-8b41-f372f69294fc-trusted-ca\") pod \"image-registry-7fc569f596-rgpxb\" (UID: \"632083f5-6ec0-4455-8b41-f372f69294fc\") " pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:10.051573 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.051545    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/632083f5-6ec0-4455-8b41-f372f69294fc-installation-pull-secrets\") pod \"image-registry-7fc569f596-rgpxb\" (UID: \"632083f5-6ec0-4455-8b41-f372f69294fc\") " pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:10.052036 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.052017    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/632083f5-6ec0-4455-8b41-f372f69294fc-image-registry-private-configuration\") pod \"image-registry-7fc569f596-rgpxb\" (UID: \"632083f5-6ec0-4455-8b41-f372f69294fc\") " pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:10.052450 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.052428    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/632083f5-6ec0-4455-8b41-f372f69294fc-registry-tls\") pod \"image-registry-7fc569f596-rgpxb\" (UID: \"632083f5-6ec0-4455-8b41-f372f69294fc\") " pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:10.059054 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.059028    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj26q\" (UniqueName: \"kubernetes.io/projected/632083f5-6ec0-4455-8b41-f372f69294fc-kube-api-access-pj26q\") pod \"image-registry-7fc569f596-rgpxb\" (UID: \"632083f5-6ec0-4455-8b41-f372f69294fc\") " pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:10.060018 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.059979    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/632083f5-6ec0-4455-8b41-f372f69294fc-bound-sa-token\") pod \"image-registry-7fc569f596-rgpxb\" (UID: \"632083f5-6ec0-4455-8b41-f372f69294fc\") " pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:10.150493 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.150414    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:10.157705 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.157329    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h4cnn" event={"ID":"bef28d6a-f451-4c22-adca-7179590804ea","Type":"ContainerStarted","Data":"32470edd732bb5c00afc40567785fc3785410b1a56ecd230724d1e5eb3e88c88"}
Jul 17 10:59:10.157705 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.157459    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-h4cnn"
Jul 17 10:59:10.159444 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.159376    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gf598" event={"ID":"bf372042-c2bc-45b7-a95d-8a90116a13a7","Type":"ContainerStarted","Data":"36e51ebdf7a191bc054c9fd2bf1b3c7c33262f0aac8210427f2aac475087dfe7"}
Jul 17 10:59:10.250503 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.249899    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:10.329331 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.328968    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-h4cnn" podStartSLOduration=65.930305799 podStartE2EDuration="1m10.328947378s" totalImagesPullingTime="4.398641579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:59:10.184115354 +0000 UTC m=+105.113654996" watchObservedRunningTime="2026-07-17 10:59:10.328947378 +0000 UTC m=+105.258487018"
Jul 17 10:59:10.329331 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.329168    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fc569f596-rgpxb"]
Jul 17 10:59:10.332617 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:59:10.332582    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod632083f5_6ec0_4455_8b41_f372f69294fc.slice/crio-eb59aea3038b10670d5634b479492cf7dc9fb96e25e1e8bf3dd6245259d4e79a WatchSource:0}: Error finding container eb59aea3038b10670d5634b479492cf7dc9fb96e25e1e8bf3dd6245259d4e79a: Status 404 returned error can't find the container with id eb59aea3038b10670d5634b479492cf7dc9fb96e25e1e8bf3dd6245259d4e79a
Jul 17 10:59:10.333379 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.333352    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:10.408384 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.408306    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:10.511714 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.511632    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:10.697306 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:10.696949    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:11.001250 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:11.001153    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:11.165223 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:11.165175    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fc569f596-rgpxb" event={"ID":"632083f5-6ec0-4455-8b41-f372f69294fc","Type":"ContainerStarted","Data":"dbc37c16f8c6541aef5004e37237a19ca8e8966912b098fac8887fc992c541c5"}
Jul 17 10:59:11.165402 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:11.165236    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fc569f596-rgpxb" event={"ID":"632083f5-6ec0-4455-8b41-f372f69294fc","Type":"ContainerStarted","Data":"eb59aea3038b10670d5634b479492cf7dc9fb96e25e1e8bf3dd6245259d4e79a"}
Jul 17 10:59:11.188180 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:11.188133    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7fc569f596-rgpxb" podStartSLOduration=2.188117098 podStartE2EDuration="2.188117098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:59:11.186979079 +0000 UTC m=+106.116518720" watchObservedRunningTime="2026-07-17 10:59:11.188117098 +0000 UTC m=+106.117656738"
Jul 17 10:59:11.719097 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:11.718106    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:11.790909 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:11.790861    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:12.170777 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:12.170748    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:13.175386 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:13.175353    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gf598" event={"ID":"bf372042-c2bc-45b7-a95d-8a90116a13a7","Type":"ContainerStarted","Data":"2fbe1afa1d66a564bb07bdde55e2524630c36322c7fe83aa22b9b0e588c5cf5f"}
Jul 17 10:59:14.586063 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:14.586026    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8f5769f-44b8-49e1-acc9-35fc5b558035-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-5xw7l\" (UID: \"f8f5769f-44b8-49e1-acc9-35fc5b558035\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5xw7l"
Jul 17 10:59:14.588548 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:14.588516    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8f5769f-44b8-49e1-acc9-35fc5b558035-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-5xw7l\" (UID: \"f8f5769f-44b8-49e1-acc9-35fc5b558035\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5xw7l"
Jul 17 10:59:14.886121 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:14.886084    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5xw7l"
Jul 17 10:59:16.316151 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:16.316121    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5xw7l"]
Jul 17 10:59:16.319078 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:59:16.319043    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f5769f_44b8_49e1_acc9_35fc5b558035.slice/crio-344dff2d18222d22fb71d002d79cec13449d9d894906dd6891797399f95eeee7 WatchSource:0}: Error finding container 344dff2d18222d22fb71d002d79cec13449d9d894906dd6891797399f95eeee7: Status 404 returned error can't find the container with id 344dff2d18222d22fb71d002d79cec13449d9d894906dd6891797399f95eeee7
Jul 17 10:59:17.191118 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:17.191083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gf598" event={"ID":"bf372042-c2bc-45b7-a95d-8a90116a13a7","Type":"ContainerStarted","Data":"a509358ea0bd99ad08159eccc9ec2f44bc1ef0f36da8a1a13d0d6c141eb49c62"}
Jul 17 10:59:17.192460 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:17.192433    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5xw7l" event={"ID":"f8f5769f-44b8-49e1-acc9-35fc5b558035","Type":"ContainerStarted","Data":"344dff2d18222d22fb71d002d79cec13449d9d894906dd6891797399f95eeee7"}
Jul 17 10:59:17.386410 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:17.386370    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:17.481409 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:17.481323    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:18.543864 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:18.543783    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:18.626629 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:18.626588    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:18.748401 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:18.748367    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:18.959909 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:18.959874    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:19.201600 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.201564    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5xw7l" event={"ID":"f8f5769f-44b8-49e1-acc9-35fc5b558035","Type":"ContainerStarted","Data":"80342607539a918f43376d06cbcc852a0e2331fdc8dbcad6c89e87a6a9b07958"}
Jul 17 10:59:19.225937 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.225839    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-5xw7l" podStartSLOduration=35.383295213 podStartE2EDuration="37.225820441s" totalImagesPullingTime="1.842525228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:59:19.22444269 +0000 UTC m=+114.153982331" watchObservedRunningTime="2026-07-17 10:59:19.225820441 +0000 UTC m=+114.155360083"
Jul 17 10:59:19.226304 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.226277    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gf598" podStartSLOduration=19.314043278 podStartE2EDuration="32.226269004s" totalImagesPullingTime="12.912225726s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 10:58:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:59:17.232858438 +0000 UTC m=+112.162398078" watchObservedRunningTime="2026-07-17 10:59:19.226269004 +0000 UTC m=+114.155808645"
Jul 17 10:59:19.315150 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.315114    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b67c7549b-8c4bq"]
Jul 17 10:59:19.318192 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.318175    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b67c7549b-8c4bq"
Jul 17 10:59:19.321112 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.320998    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 17 10:59:19.321112 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.321009    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 17 10:59:19.321112 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.321027    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 17 10:59:19.321112 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.321052    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 17 10:59:19.321112 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.321098    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tnv9t\""
Jul 17 10:59:19.321446 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.321283    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 17 10:59:19.321446 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.321312    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 17 10:59:19.321446 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.321438    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 17 10:59:19.328614 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.328592    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b67c7549b-8c4bq"]
Jul 17 10:59:19.428326 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.428297    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d4261da-e6d0-40f1-a212-606204bdd893-service-ca\") pod \"console-6b67c7549b-8c4bq\" (UID: \"5d4261da-e6d0-40f1-a212-606204bdd893\") " pod="openshift-console/console-6b67c7549b-8c4bq"
Jul 17 10:59:19.428460 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.428354    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d4261da-e6d0-40f1-a212-606204bdd893-console-serving-cert\") pod \"console-6b67c7549b-8c4bq\" (UID: \"5d4261da-e6d0-40f1-a212-606204bdd893\") " pod="openshift-console/console-6b67c7549b-8c4bq"
Jul 17 10:59:19.428460 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.428416    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6mzw\" (UniqueName: \"kubernetes.io/projected/5d4261da-e6d0-40f1-a212-606204bdd893-kube-api-access-f6mzw\") pod \"console-6b67c7549b-8c4bq\" (UID: \"5d4261da-e6d0-40f1-a212-606204bdd893\") " pod="openshift-console/console-6b67c7549b-8c4bq"
Jul 17 10:59:19.428550 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.428462    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d4261da-e6d0-40f1-a212-606204bdd893-oauth-serving-cert\") pod \"console-6b67c7549b-8c4bq\" (UID: \"5d4261da-e6d0-40f1-a212-606204bdd893\") " pod="openshift-console/console-6b67c7549b-8c4bq"
Jul 17 10:59:19.428550 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.428481    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d4261da-e6d0-40f1-a212-606204bdd893-console-oauth-config\") pod \"console-6b67c7549b-8c4bq\" (UID: \"5d4261da-e6d0-40f1-a212-606204bdd893\") " pod="openshift-console/console-6b67c7549b-8c4bq"
Jul 17 10:59:19.428550 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.428520    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d4261da-e6d0-40f1-a212-606204bdd893-console-config\") pod \"console-6b67c7549b-8c4bq\" (UID: \"5d4261da-e6d0-40f1-a212-606204bdd893\") " pod="openshift-console/console-6b67c7549b-8c4bq"
Jul 17 10:59:19.529682 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.529594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d4261da-e6d0-40f1-a212-606204bdd893-console-config\") pod \"console-6b67c7549b-8c4bq\" (UID: \"5d4261da-e6d0-40f1-a212-606204bdd893\") " pod="openshift-console/console-6b67c7549b-8c4bq"
Jul 17 10:59:19.529682 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.529634    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d4261da-e6d0-40f1-a212-606204bdd893-service-ca\") pod \"console-6b67c7549b-8c4bq\" (UID: \"5d4261da-e6d0-40f1-a212-606204bdd893\") " pod="openshift-console/console-6b67c7549b-8c4bq"
Jul 17 10:59:19.529890 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.529687    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d4261da-e6d0-40f1-a212-606204bdd893-console-serving-cert\") pod \"console-6b67c7549b-8c4bq\" (UID: \"5d4261da-e6d0-40f1-a212-606204bdd893\") " pod="openshift-console/console-6b67c7549b-8c4bq"
Jul 17 10:59:19.529890 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.529717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6mzw\" (UniqueName: \"kubernetes.io/projected/5d4261da-e6d0-40f1-a212-606204bdd893-kube-api-access-f6mzw\") pod \"console-6b67c7549b-8c4bq\" (UID: \"5d4261da-e6d0-40f1-a212-606204bdd893\") " pod="openshift-console/console-6b67c7549b-8c4bq"
Jul 17 10:59:19.529890 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.529740    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d4261da-e6d0-40f1-a212-606204bdd893-oauth-serving-cert\") pod \"console-6b67c7549b-8c4bq\" (UID: \"5d4261da-e6d0-40f1-a212-606204bdd893\") " pod="openshift-console/console-6b67c7549b-8c4bq"
Jul 17 10:59:19.529890 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.529768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d4261da-e6d0-40f1-a212-606204bdd893-console-oauth-config\") pod \"console-6b67c7549b-8c4bq\" (UID: \"5d4261da-e6d0-40f1-a212-606204bdd893\") " pod="openshift-console/console-6b67c7549b-8c4bq"
Jul 17 10:59:19.530456 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.530433    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d4261da-e6d0-40f1-a212-606204bdd893-console-config\") pod \"console-6b67c7549b-8c4bq\" (UID: \"5d4261da-e6d0-40f1-a212-606204bdd893\") " pod="openshift-console/console-6b67c7549b-8c4bq"
Jul 17 10:59:19.530456 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.530443    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d4261da-e6d0-40f1-a212-606204bdd893-service-ca\") pod \"console-6b67c7549b-8c4bq\" (UID: \"5d4261da-e6d0-40f1-a212-606204bdd893\") " pod="openshift-console/console-6b67c7549b-8c4bq"
Jul 17 10:59:19.530604 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.530555    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d4261da-e6d0-40f1-a212-606204bdd893-oauth-serving-cert\") pod \"console-6b67c7549b-8c4bq\" (UID: \"5d4261da-e6d0-40f1-a212-606204bdd893\") " pod="openshift-console/console-6b67c7549b-8c4bq"
Jul 17 10:59:19.532096 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.532053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d4261da-e6d0-40f1-a212-606204bdd893-console-oauth-config\") pod \"console-6b67c7549b-8c4bq\" (UID: \"5d4261da-e6d0-40f1-a212-606204bdd893\") " pod="openshift-console/console-6b67c7549b-8c4bq"
Jul 17 10:59:19.532178 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.532107    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d4261da-e6d0-40f1-a212-606204bdd893-console-serving-cert\") pod \"console-6b67c7549b-8c4bq\" (UID: \"5d4261da-e6d0-40f1-a212-606204bdd893\") " pod="openshift-console/console-6b67c7549b-8c4bq"
Jul 17 10:59:19.542594 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.542562    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6mzw\" (UniqueName: \"kubernetes.io/projected/5d4261da-e6d0-40f1-a212-606204bdd893-kube-api-access-f6mzw\") pod \"console-6b67c7549b-8c4bq\" (UID: \"5d4261da-e6d0-40f1-a212-606204bdd893\") " pod="openshift-console/console-6b67c7549b-8c4bq"
Jul 17 10:59:19.627792 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.627735    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b67c7549b-8c4bq"
Jul 17 10:59:19.755473 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.755445    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b67c7549b-8c4bq"]
Jul 17 10:59:19.758291 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:59:19.758260    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d4261da_e6d0_40f1_a212_606204bdd893.slice/crio-e2e9ae1a2195f05bef34e0190fef58fe9324dd9c32585226ec40f4adbd722079 WatchSource:0}: Error finding container e2e9ae1a2195f05bef34e0190fef58fe9324dd9c32585226ec40f4adbd722079: Status 404 returned error can't find the container with id e2e9ae1a2195f05bef34e0190fef58fe9324dd9c32585226ec40f4adbd722079
Jul 17 10:59:19.782401 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.782333    2563 patch_prober.go:28] interesting pod/image-registry-66458c5c5-xdg6l container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 17 10:59:19.782401 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:19.782383    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-66458c5c5-xdg6l" podUID="55297f57-dfcc-4655-87ae-ca25456a8265" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 10:59:20.167160 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:20.167134    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-h4cnn"
Jul 17 10:59:20.209950 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:20.209901    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b67c7549b-8c4bq" event={"ID":"5d4261da-e6d0-40f1-a212-606204bdd893","Type":"ContainerStarted","Data":"e2e9ae1a2195f05bef34e0190fef58fe9324dd9c32585226ec40f4adbd722079"}
Jul 17 10:59:23.221172 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:23.221130    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b67c7549b-8c4bq" event={"ID":"5d4261da-e6d0-40f1-a212-606204bdd893","Type":"ContainerStarted","Data":"d3f835a9e2200c2f9756bb935657e70b402c5f7436260ba37aded999b57c230b"}
Jul 17 10:59:23.835794 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:23.835764    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:23.938515 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:23.938488    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:24.698263 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:24.698231    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:24.756964 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:24.756931    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:28.369388 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:28.369356    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:28.462858 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:28.462823    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:28.568655 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:28.568621    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:28.750579 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:28.750506    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:28.852013 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:28.851980    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:29.147569 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:29.147538    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:29.276217 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:29.276177    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:29.628465 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:29.628375    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b67c7549b-8c4bq"
Jul 17 10:59:29.628465 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:29.628438    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b67c7549b-8c4bq"
Jul 17 10:59:29.634115 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:29.634091    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b67c7549b-8c4bq"
Jul 17 10:59:29.655917 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:29.655862    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b67c7549b-8c4bq" podStartSLOduration=7.568634018 podStartE2EDuration="10.655846336s" totalImagesPullingTime="3.087212318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:59:23.240987742 +0000 UTC m=+118.170527381" watchObservedRunningTime="2026-07-17 10:59:29.655846336 +0000 UTC m=+124.585385979"
Jul 17 10:59:29.782621 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:29.782592    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:59:30.138355 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:30.138323    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:30.242610 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:30.242574    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:30.248437 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:30.248414    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b67c7549b-8c4bq"
Jul 17 10:59:31.777204 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:31.777172    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:32.950495 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:32.950463    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:33.032392 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:33.032361    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:33.179372 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:33.179343    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7fc569f596-rgpxb"
Jul 17 10:59:33.335807 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:33.335702    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:33.434565 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:33.434532    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:33.535599 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:33.535554    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:33.634603 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:33.634572    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:33.929728 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:33.929627    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:34.160631 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.160552    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:34.181537 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.181455    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-gmdhf"]
Jul 17 10:59:34.186264 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.186242    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gmdhf"
Jul 17 10:59:34.188818 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.188798    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 17 10:59:34.188992 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.188821    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 17 10:59:34.189060 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.188849    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-n6sdp\""
Jul 17 10:59:34.189939 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.189921    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 17 10:59:34.195667 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.195644    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-gmdhf"]
Jul 17 10:59:34.199970 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.199943    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gs7hc"]
Jul 17 10:59:34.203424 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.203400    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.205817 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.205636    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tpqvh\""
Jul 17 10:59:34.205817 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.205726    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 17 10:59:34.205989 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.205972    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 17 10:59:34.206438 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.206281    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 17 10:59:34.237143 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.237111    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-gmdhf\" (UID: \"e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gmdhf"
Jul 17 10:59:34.237305 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.237154    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-sys\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.237305 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.237190    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9wmb\" (UniqueName: \"kubernetes.io/projected/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-kube-api-access-p9wmb\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.237305 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.237219    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.237305 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.237269    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-node-exporter-accelerators-collector-config\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.237305 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.237300    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-node-exporter-tls\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.237541 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.237332    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-gmdhf\" (UID: \"e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gmdhf"
Jul 17 10:59:34.237541 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.237354    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-root\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.237541 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.237385    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-gmdhf\" (UID: \"e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gmdhf"
Jul 17 10:59:34.237541 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.237446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-node-exporter-wtmp\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.237541 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.237497    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9c3cb80-365e-4c10-bde0-1e9f17948a97-metrics-certs\") pod \"network-metrics-daemon-s5c9b\" (UID: \"e9c3cb80-365e-4c10-bde0-1e9f17948a97\") " pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:59:34.237541 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.237528    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m56gr\" (UniqueName: \"kubernetes.io/projected/e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0-kube-api-access-m56gr\") pod \"openshift-state-metrics-79f6958cdf-gmdhf\" (UID: \"e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gmdhf"
Jul 17 10:59:34.237818 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.237550    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-node-exporter-textfile\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.237818 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.237571    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-metrics-client-ca\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.240423 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.240392    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9c3cb80-365e-4c10-bde0-1e9f17948a97-metrics-certs\") pod \"network-metrics-daemon-s5c9b\" (UID: \"e9c3cb80-365e-4c10-bde0-1e9f17948a97\") " pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:59:34.248170 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.247998    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:34.317578 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.317538    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:34.338828 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.338789    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-node-exporter-accelerators-collector-config\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.339020 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.338846    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-node-exporter-tls\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.339020 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.338880    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-gmdhf\" (UID: \"e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gmdhf"
Jul 17 10:59:34.339020 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.338907    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-root\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.339020 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.338940    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-gmdhf\" (UID: \"e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gmdhf"
Jul 17 10:59:34.339020 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.338975    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-node-exporter-wtmp\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.339020 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.339015    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m56gr\" (UniqueName: \"kubernetes.io/projected/e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0-kube-api-access-m56gr\") pod \"openshift-state-metrics-79f6958cdf-gmdhf\" (UID: \"e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gmdhf"
Jul 17 10:59:34.339342 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.339039    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-node-exporter-textfile\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.339342 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.339062    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-metrics-client-ca\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.339342 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.339118    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-gmdhf\" (UID: \"e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gmdhf"
Jul 17 10:59:34.339342 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.339142    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-sys\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.339342 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.339174    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9wmb\" (UniqueName: \"kubernetes.io/projected/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-kube-api-access-p9wmb\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.339342 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.339202    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.340753 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.339733    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-root\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.340753 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.340053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-gmdhf\" (UID: \"e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gmdhf"
Jul 17 10:59:34.340753 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:59:34.340183    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 17 10:59:34.340753 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.340210    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-metrics-client-ca\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.340753 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:59:34.340254    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-node-exporter-tls podName:c8e6f422-93f9-49cb-8ff3-cad07c26c9e4 nodeName:}" failed. No retries permitted until 2026-07-17 10:59:34.840221888 +0000 UTC m=+129.769761509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-node-exporter-tls") pod "node-exporter-gs7hc" (UID: "c8e6f422-93f9-49cb-8ff3-cad07c26c9e4") : secret "node-exporter-tls" not found
Jul 17 10:59:34.340753 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.340453    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-node-exporter-textfile\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.340753 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.340494    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-sys\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.340753 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.340494    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-node-exporter-accelerators-collector-config\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.340753 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:59:34.340568    2563 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 17 10:59:34.340753 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:59:34.340604    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0-openshift-state-metrics-tls podName:e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0 nodeName:}" failed. No retries permitted until 2026-07-17 10:59:34.840592206 +0000 UTC m=+129.770131832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-gmdhf" (UID: "e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0") : secret "openshift-state-metrics-tls" not found
Jul 17 10:59:34.340753 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.340747    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-node-exporter-wtmp\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.343141 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.343112    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.349415 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.349373    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m56gr\" (UniqueName: \"kubernetes.io/projected/e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0-kube-api-access-m56gr\") pod \"openshift-state-metrics-79f6958cdf-gmdhf\" (UID: \"e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gmdhf"
Jul 17 10:59:34.370561 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.370523    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-gmdhf\" (UID: \"e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gmdhf"
Jul 17 10:59:34.377675 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.377610    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9wmb\" (UniqueName: \"kubernetes.io/projected/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-kube-api-access-p9wmb\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.386609 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.385816    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m4ccj\""
Jul 17 10:59:34.390277 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.390236    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c9b"
Jul 17 10:59:34.483111 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.481954    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:34.567414 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.567377    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s5c9b"]
Jul 17 10:59:34.570486 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:59:34.570448    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c3cb80_365e_4c10_bde0_1e9f17948a97.slice/crio-978fe98f8c17eaf009c0b86d82a74c490154bb450ddc0ee20aab1bd9d13343ce WatchSource:0}: Error finding container 978fe98f8c17eaf009c0b86d82a74c490154bb450ddc0ee20aab1bd9d13343ce: Status 404 returned error can't find the container with id 978fe98f8c17eaf009c0b86d82a74c490154bb450ddc0ee20aab1bd9d13343ce
Jul 17 10:59:34.575644 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.575608    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:34.646656 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.646608    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:34.748604 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.748519    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:34.807902 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.807862    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-66458c5c5-xdg6l" podUID="55297f57-dfcc-4655-87ae-ca25456a8265" containerName="registry" containerID="cri-o://ff7cc561a615628fe7cf2112d8f0b9191ebea323c863c4369e6d0f77ae101225" gracePeriod=30
Jul 17 10:59:34.842923 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.842894    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-node-exporter-tls\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.843060 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.842950    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-gmdhf\" (UID: \"e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gmdhf"
Jul 17 10:59:34.845287 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.845269    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c8e6f422-93f9-49cb-8ff3-cad07c26c9e4-node-exporter-tls\") pod \"node-exporter-gs7hc\" (UID: \"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4\") " pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:34.845445 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.845425    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-gmdhf\" (UID: \"e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gmdhf"
Jul 17 10:59:34.846690 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:34.846664    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:35.009938 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.009842    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:35.044685 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.044664    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:59:35.097518 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.097477    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gmdhf"
Jul 17 10:59:35.121500 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.121470    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gs7hc"
Jul 17 10:59:35.134160 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:59:35.134128    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e6f422_93f9_49cb_8ff3_cad07c26c9e4.slice/crio-32c0a728fc175545524157b36e6406d84732bc405bb24978b5d45e39cf0d94e9 WatchSource:0}: Error finding container 32c0a728fc175545524157b36e6406d84732bc405bb24978b5d45e39cf0d94e9: Status 404 returned error can't find the container with id 32c0a728fc175545524157b36e6406d84732bc405bb24978b5d45e39cf0d94e9
Jul 17 10:59:35.146191 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.145428    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qwks\" (UniqueName: \"kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-kube-api-access-6qwks\") pod \"55297f57-dfcc-4655-87ae-ca25456a8265\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") "
Jul 17 10:59:35.146191 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.145469    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/55297f57-dfcc-4655-87ae-ca25456a8265-trusted-ca\") pod \"55297f57-dfcc-4655-87ae-ca25456a8265\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") "
Jul 17 10:59:35.146191 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.145522    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-bound-sa-token\") pod \"55297f57-dfcc-4655-87ae-ca25456a8265\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") "
Jul 17 10:59:35.146191 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.145549    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/55297f57-dfcc-4655-87ae-ca25456a8265-ca-trust-extracted\") pod \"55297f57-dfcc-4655-87ae-ca25456a8265\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") "
Jul 17 10:59:35.146191 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.145584    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/55297f57-dfcc-4655-87ae-ca25456a8265-registry-certificates\") pod \"55297f57-dfcc-4655-87ae-ca25456a8265\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") "
Jul 17 10:59:35.146191 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.145618    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/55297f57-dfcc-4655-87ae-ca25456a8265-installation-pull-secrets\") pod \"55297f57-dfcc-4655-87ae-ca25456a8265\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") "
Jul 17 10:59:35.146191 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.145661    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-registry-tls\") pod \"55297f57-dfcc-4655-87ae-ca25456a8265\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") "
Jul 17 10:59:35.146191 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.145700    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/55297f57-dfcc-4655-87ae-ca25456a8265-image-registry-private-configuration\") pod \"55297f57-dfcc-4655-87ae-ca25456a8265\" (UID: \"55297f57-dfcc-4655-87ae-ca25456a8265\") "
Jul 17 10:59:35.151370 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.148221    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55297f57-dfcc-4655-87ae-ca25456a8265-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "55297f57-dfcc-4655-87ae-ca25456a8265" (UID: "55297f57-dfcc-4655-87ae-ca25456a8265"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 10:59:35.151370 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.147847    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55297f57-dfcc-4655-87ae-ca25456a8265-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "55297f57-dfcc-4655-87ae-ca25456a8265" (UID: "55297f57-dfcc-4655-87ae-ca25456a8265"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 10:59:35.151853 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.151768    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "55297f57-dfcc-4655-87ae-ca25456a8265" (UID: "55297f57-dfcc-4655-87ae-ca25456a8265"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 10:59:35.151853 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.151828    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "55297f57-dfcc-4655-87ae-ca25456a8265" (UID: "55297f57-dfcc-4655-87ae-ca25456a8265"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 10:59:35.152434 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.152393    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55297f57-dfcc-4655-87ae-ca25456a8265-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "55297f57-dfcc-4655-87ae-ca25456a8265" (UID: "55297f57-dfcc-4655-87ae-ca25456a8265"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 10:59:35.152724 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.152688    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55297f57-dfcc-4655-87ae-ca25456a8265-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "55297f57-dfcc-4655-87ae-ca25456a8265" (UID: "55297f57-dfcc-4655-87ae-ca25456a8265"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 10:59:35.153478 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.153427    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-kube-api-access-6qwks" (OuterVolumeSpecName: "kube-api-access-6qwks") pod "55297f57-dfcc-4655-87ae-ca25456a8265" (UID: "55297f57-dfcc-4655-87ae-ca25456a8265"). InnerVolumeSpecName "kube-api-access-6qwks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 10:59:35.164454 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.164422    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55297f57-dfcc-4655-87ae-ca25456a8265-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "55297f57-dfcc-4655-87ae-ca25456a8265" (UID: "55297f57-dfcc-4655-87ae-ca25456a8265"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 10:59:35.191619 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.191586    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:35.246998 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.246793    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/55297f57-dfcc-4655-87ae-ca25456a8265-registry-certificates\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 10:59:35.246998 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.246827    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/55297f57-dfcc-4655-87ae-ca25456a8265-installation-pull-secrets\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 10:59:35.246998 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.246842    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-registry-tls\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 10:59:35.246998 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.246856    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/55297f57-dfcc-4655-87ae-ca25456a8265-image-registry-private-configuration\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 10:59:35.246998 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.246871    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qwks\" (UniqueName: \"kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-kube-api-access-6qwks\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 10:59:35.246998 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.246884    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/55297f57-dfcc-4655-87ae-ca25456a8265-trusted-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 10:59:35.246998 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.246896    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55297f57-dfcc-4655-87ae-ca25456a8265-bound-sa-token\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 10:59:35.246998 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.246909    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/55297f57-dfcc-4655-87ae-ca25456a8265-ca-trust-extracted\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 10:59:35.264168 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.264019    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-gmdhf"]
Jul 17 10:59:35.265773 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.265746    2563 generic.go:356] "Generic (PLEG): container finished" podID="55297f57-dfcc-4655-87ae-ca25456a8265" containerID="ff7cc561a615628fe7cf2112d8f0b9191ebea323c863c4369e6d0f77ae101225" exitCode=0
Jul 17 10:59:35.265912 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.265866    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66458c5c5-xdg6l" event={"ID":"55297f57-dfcc-4655-87ae-ca25456a8265","Type":"ContainerDied","Data":"ff7cc561a615628fe7cf2112d8f0b9191ebea323c863c4369e6d0f77ae101225"}
Jul 17 10:59:35.265912 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.265907    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66458c5c5-xdg6l" event={"ID":"55297f57-dfcc-4655-87ae-ca25456a8265","Type":"ContainerDied","Data":"38f05e1dfe4afb602f1df8cad57efea82a9e87cf987ce91931d67058e2c6fe2c"}
Jul 17 10:59:35.266031 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.265937    2563 scope.go:117] "RemoveContainer" containerID="ff7cc561a615628fe7cf2112d8f0b9191ebea323c863c4369e6d0f77ae101225"
Jul 17 10:59:35.266225 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.266138    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66458c5c5-xdg6l"
Jul 17 10:59:35.269146 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.269043    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gs7hc" event={"ID":"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4","Type":"ContainerStarted","Data":"32c0a728fc175545524157b36e6406d84732bc405bb24978b5d45e39cf0d94e9"}
Jul 17 10:59:35.271438 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.271044    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s5c9b" event={"ID":"e9c3cb80-365e-4c10-bde0-1e9f17948a97","Type":"ContainerStarted","Data":"978fe98f8c17eaf009c0b86d82a74c490154bb450ddc0ee20aab1bd9d13343ce"}
Jul 17 10:59:35.283135 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.283114    2563 scope.go:117] "RemoveContainer" containerID="ff7cc561a615628fe7cf2112d8f0b9191ebea323c863c4369e6d0f77ae101225"
Jul 17 10:59:35.283796 ip-10-0-140-233 kubenswrapper[2563]: E0717 10:59:35.283709    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff7cc561a615628fe7cf2112d8f0b9191ebea323c863c4369e6d0f77ae101225\": container with ID starting with ff7cc561a615628fe7cf2112d8f0b9191ebea323c863c4369e6d0f77ae101225 not found: ID does not exist" containerID="ff7cc561a615628fe7cf2112d8f0b9191ebea323c863c4369e6d0f77ae101225"
Jul 17 10:59:35.283796 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.283751    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff7cc561a615628fe7cf2112d8f0b9191ebea323c863c4369e6d0f77ae101225"} err="failed to get container status \"ff7cc561a615628fe7cf2112d8f0b9191ebea323c863c4369e6d0f77ae101225\": rpc error: code = NotFound desc = could not find container \"ff7cc561a615628fe7cf2112d8f0b9191ebea323c863c4369e6d0f77ae101225\": container with ID starting with ff7cc561a615628fe7cf2112d8f0b9191ebea323c863c4369e6d0f77ae101225 not found: ID does not exist"
Jul 17 10:59:35.289463 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.289407    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:35.298677 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.298649    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66458c5c5-xdg6l"]
Jul 17 10:59:35.309562 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.309533    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-66458c5c5-xdg6l"]
Jul 17 10:59:35.369118 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.369089    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:35.663512 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:35.663480    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55297f57-dfcc-4655-87ae-ca25456a8265" path="/var/lib/kubelet/pods/55297f57-dfcc-4655-87ae-ca25456a8265/volumes"
Jul 17 10:59:36.192769 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.192728    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:36.276798 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.276758    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gmdhf" event={"ID":"e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0","Type":"ContainerStarted","Data":"fbf2d7bbbe900ccb99b5c0e4049b8f6c349ccec853fec17f8d57a3935586f630"}
Jul 17 10:59:36.276953 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.276865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gmdhf" event={"ID":"e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0","Type":"ContainerStarted","Data":"92b095f99307b3e6632a7eb7b0fc9e6257d73a0feb8da1c9db00b0b620ff45d8"}
Jul 17 10:59:36.276953 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.276896    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gmdhf" event={"ID":"e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0","Type":"ContainerStarted","Data":"d067c9c489cea42a90d807b91a0b3e45112b05e7f1238378304fe70bb5cb25ef"}
Jul 17 10:59:36.279742 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.279719    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s5c9b" event={"ID":"e9c3cb80-365e-4c10-bde0-1e9f17948a97","Type":"ContainerStarted","Data":"5142156f160ea7f8c05aba5a60045ee4af85a3b6a32265325888476b1ee736dd"}
Jul 17 10:59:36.279863 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.279760    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s5c9b" event={"ID":"e9c3cb80-365e-4c10-bde0-1e9f17948a97","Type":"ContainerStarted","Data":"03dcb646c549be5748bd27451eebe25ec58d1deb66b3970bc9e5fc4c5923406b"}
Jul 17 10:59:36.324657 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.324591    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-s5c9b" podStartSLOduration=129.224567714 podStartE2EDuration="2m10.324575641s" totalImagesPullingTime="1.100007927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:59:36.32301899 +0000 UTC m=+131.252558631" watchObservedRunningTime="2026-07-17 10:59:36.324575641 +0000 UTC m=+131.254115328"
Jul 17 10:59:36.327592 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.327564    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:36.340332 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.340299    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-78cd79d867-5lv8s"]
Jul 17 10:59:36.340590 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.340573    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55297f57-dfcc-4655-87ae-ca25456a8265" containerName="registry"
Jul 17 10:59:36.340680 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.340592    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="55297f57-dfcc-4655-87ae-ca25456a8265" containerName="registry"
Jul 17 10:59:36.340680 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.340675    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="55297f57-dfcc-4655-87ae-ca25456a8265" containerName="registry"
Jul 17 10:59:36.344291 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.344271    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:36.347521 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.347491    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 17 10:59:36.347633 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.347524    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5zx8j\""
Jul 17 10:59:36.347633 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.347597    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 17 10:59:36.347751 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.347738    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-a1hojg085824q\""
Jul 17 10:59:36.347801 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.347784    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 17 10:59:36.347865 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.347848    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 17 10:59:36.348001 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.347971    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 17 10:59:36.366998 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.366964    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-78cd79d867-5lv8s"]
Jul 17 10:59:36.455465 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.455442    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzlx4\" (UniqueName: \"kubernetes.io/projected/f440b569-a68f-48e6-8a50-51b48d4e71e1-kube-api-access-lzlx4\") pod \"thanos-querier-78cd79d867-5lv8s\" (UID: \"f440b569-a68f-48e6-8a50-51b48d4e71e1\") " pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:36.455560 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.455483    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f440b569-a68f-48e6-8a50-51b48d4e71e1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78cd79d867-5lv8s\" (UID: \"f440b569-a68f-48e6-8a50-51b48d4e71e1\") " pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:36.455560 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.455507    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f440b569-a68f-48e6-8a50-51b48d4e71e1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78cd79d867-5lv8s\" (UID: \"f440b569-a68f-48e6-8a50-51b48d4e71e1\") " pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:36.455560 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.455549    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f440b569-a68f-48e6-8a50-51b48d4e71e1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78cd79d867-5lv8s\" (UID: \"f440b569-a68f-48e6-8a50-51b48d4e71e1\") " pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:36.455658 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.455566    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f440b569-a68f-48e6-8a50-51b48d4e71e1-metrics-client-ca\") pod \"thanos-querier-78cd79d867-5lv8s\" (UID: \"f440b569-a68f-48e6-8a50-51b48d4e71e1\") " pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:36.455658 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.455626    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f440b569-a68f-48e6-8a50-51b48d4e71e1-secret-grpc-tls\") pod \"thanos-querier-78cd79d867-5lv8s\" (UID: \"f440b569-a68f-48e6-8a50-51b48d4e71e1\") " pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:36.455658 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.455652    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f440b569-a68f-48e6-8a50-51b48d4e71e1-secret-thanos-querier-tls\") pod \"thanos-querier-78cd79d867-5lv8s\" (UID: \"f440b569-a68f-48e6-8a50-51b48d4e71e1\") " pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:36.455745 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.455686    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f440b569-a68f-48e6-8a50-51b48d4e71e1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78cd79d867-5lv8s\" (UID: \"f440b569-a68f-48e6-8a50-51b48d4e71e1\") " pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:36.472399 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.472372    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:36.556712 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.556684    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f440b569-a68f-48e6-8a50-51b48d4e71e1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78cd79d867-5lv8s\" (UID: \"f440b569-a68f-48e6-8a50-51b48d4e71e1\") " pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:36.556863 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.556720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f440b569-a68f-48e6-8a50-51b48d4e71e1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78cd79d867-5lv8s\" (UID: \"f440b569-a68f-48e6-8a50-51b48d4e71e1\") " pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:36.556863 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.556748    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f440b569-a68f-48e6-8a50-51b48d4e71e1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78cd79d867-5lv8s\" (UID: \"f440b569-a68f-48e6-8a50-51b48d4e71e1\") " pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:36.556863 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.556773    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f440b569-a68f-48e6-8a50-51b48d4e71e1-metrics-client-ca\") pod \"thanos-querier-78cd79d867-5lv8s\" (UID: \"f440b569-a68f-48e6-8a50-51b48d4e71e1\") " pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:36.557042 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.556951    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f440b569-a68f-48e6-8a50-51b48d4e71e1-secret-grpc-tls\") pod \"thanos-querier-78cd79d867-5lv8s\" (UID: \"f440b569-a68f-48e6-8a50-51b48d4e71e1\") " pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:36.557042 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.557008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f440b569-a68f-48e6-8a50-51b48d4e71e1-secret-thanos-querier-tls\") pod \"thanos-querier-78cd79d867-5lv8s\" (UID: \"f440b569-a68f-48e6-8a50-51b48d4e71e1\") " pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:36.557172 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.557097    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f440b569-a68f-48e6-8a50-51b48d4e71e1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78cd79d867-5lv8s\" (UID: \"f440b569-a68f-48e6-8a50-51b48d4e71e1\") " pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:36.557172 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.557151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzlx4\" (UniqueName: \"kubernetes.io/projected/f440b569-a68f-48e6-8a50-51b48d4e71e1-kube-api-access-lzlx4\") pod \"thanos-querier-78cd79d867-5lv8s\" (UID: \"f440b569-a68f-48e6-8a50-51b48d4e71e1\") " pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:36.557996 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.557966    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f440b569-a68f-48e6-8a50-51b48d4e71e1-metrics-client-ca\") pod \"thanos-querier-78cd79d867-5lv8s\" (UID: \"f440b569-a68f-48e6-8a50-51b48d4e71e1\") " pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:36.559595 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.559570    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f440b569-a68f-48e6-8a50-51b48d4e71e1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78cd79d867-5lv8s\" (UID: \"f440b569-a68f-48e6-8a50-51b48d4e71e1\") " pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:36.559936 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.559906    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f440b569-a68f-48e6-8a50-51b48d4e71e1-secret-thanos-querier-tls\") pod \"thanos-querier-78cd79d867-5lv8s\" (UID: \"f440b569-a68f-48e6-8a50-51b48d4e71e1\") " pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:36.560092 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.560024    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f440b569-a68f-48e6-8a50-51b48d4e71e1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78cd79d867-5lv8s\" (UID: \"f440b569-a68f-48e6-8a50-51b48d4e71e1\") " pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:36.560178 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.560100    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f440b569-a68f-48e6-8a50-51b48d4e71e1-secret-grpc-tls\") pod \"thanos-querier-78cd79d867-5lv8s\" (UID: \"f440b569-a68f-48e6-8a50-51b48d4e71e1\") " pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:36.560535 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.560309    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f440b569-a68f-48e6-8a50-51b48d4e71e1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78cd79d867-5lv8s\" (UID: \"f440b569-a68f-48e6-8a50-51b48d4e71e1\") " pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:36.561605 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.561574    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f440b569-a68f-48e6-8a50-51b48d4e71e1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78cd79d867-5lv8s\" (UID: \"f440b569-a68f-48e6-8a50-51b48d4e71e1\") " pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:36.566769 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.566721    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzlx4\" (UniqueName: \"kubernetes.io/projected/f440b569-a68f-48e6-8a50-51b48d4e71e1-kube-api-access-lzlx4\") pod \"thanos-querier-78cd79d867-5lv8s\" (UID: \"f440b569-a68f-48e6-8a50-51b48d4e71e1\") " pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:36.573682 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.573644    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:36.655133 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.655098    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:36.790900 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:36.790855    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-78cd79d867-5lv8s"]
Jul 17 10:59:36.794675 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:59:36.794649    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf440b569_a68f_48e6_8a50_51b48d4e71e1.slice/crio-b2450e7706422ccb70e5f2266f3a53f560c988a064cdf650f9741935762bf162 WatchSource:0}: Error finding container b2450e7706422ccb70e5f2266f3a53f560c988a064cdf650f9741935762bf162: Status 404 returned error can't find the container with id b2450e7706422ccb70e5f2266f3a53f560c988a064cdf650f9741935762bf162
Jul 17 10:59:37.218304 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:37.218265    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:37.285040 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:37.285001    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s" event={"ID":"f440b569-a68f-48e6-8a50-51b48d4e71e1","Type":"ContainerStarted","Data":"b2450e7706422ccb70e5f2266f3a53f560c988a064cdf650f9741935762bf162"}
Jul 17 10:59:37.286774 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:37.286749    2563 generic.go:356] "Generic (PLEG): container finished" podID="c8e6f422-93f9-49cb-8ff3-cad07c26c9e4" containerID="4063970eca3d776dd7e4a2b79da30872909626b480a1fe237fc4017d52fb1e29" exitCode=0
Jul 17 10:59:37.286933 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:37.286821    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gs7hc" event={"ID":"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4","Type":"ContainerDied","Data":"4063970eca3d776dd7e4a2b79da30872909626b480a1fe237fc4017d52fb1e29"}
Jul 17 10:59:37.289131 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:37.289105    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gmdhf" event={"ID":"e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0","Type":"ContainerStarted","Data":"97002eb9725de65bd478f3fb696bb44a8f19a6834e2785d6d3aba1b9689cc486"}
Jul 17 10:59:37.311242 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:37.311201    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:37.326143 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:37.326095    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gmdhf" podStartSLOduration=2.23507324 podStartE2EDuration="3.326057931s" totalImagesPullingTime="1.090984691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:59:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:59:37.32483541 +0000 UTC m=+132.254375052" watchObservedRunningTime="2026-07-17 10:59:37.326057931 +0000 UTC m=+132.255597571"
Jul 17 10:59:37.375533 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:37.375501    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:37.609598 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:37.609510    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:38.296003 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.295965    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gs7hc" event={"ID":"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4","Type":"ContainerStarted","Data":"c0e321976a91cb218944f995aa3a1457ce51e7dae7da23dd566bd417d6965630"}
Jul 17 10:59:38.296507 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.296033    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gs7hc" event={"ID":"c8e6f422-93f9-49cb-8ff3-cad07c26c9e4","Type":"ContainerStarted","Data":"9d29cda175ec75bab46ff7049ff1295469ebf9054f5b482cf1f2f18edf3300c7"}
Jul 17 10:59:38.327893 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.327827    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gs7hc" podStartSLOduration=2.943024567 podStartE2EDuration="4.327775432s" totalImagesPullingTime="1.315687913s" totalInitContainerRuntime="69.062952ms" isStatefulPod=true podCreationTimestamp="2026-07-17 10:59:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:59:38.32560474 +0000 UTC m=+133.255144379" watchObservedRunningTime="2026-07-17 10:59:38.327775432 +0000 UTC m=+133.257315075"
Jul 17 10:59:38.437945 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.437906    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:38.524744 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.524694    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:38.525469 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.525404    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7f4c698489-2952v"]
Jul 17 10:59:38.537653 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.537630    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f4c698489-2952v"
Jul 17 10:59:38.540254 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.540222    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 17 10:59:38.540371 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.540287    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-qshj8\""
Jul 17 10:59:38.540438 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.540376    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 17 10:59:38.540786 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.540600    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 17 10:59:38.540786 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.540611    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 17 10:59:38.540786 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.540725    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-d9kpka9j15jvh\""
Jul 17 10:59:38.540980 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.540910    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f4c698489-2952v"]
Jul 17 10:59:38.574289 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.574190    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/351ba54a-209f-4ec6-8f6a-56bbef8f3e68-client-ca-bundle\") pod \"metrics-server-7f4c698489-2952v\" (UID: \"351ba54a-209f-4ec6-8f6a-56bbef8f3e68\") " pod="openshift-monitoring/metrics-server-7f4c698489-2952v"
Jul 17 10:59:38.574289 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.574252    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/351ba54a-209f-4ec6-8f6a-56bbef8f3e68-secret-metrics-server-tls\") pod \"metrics-server-7f4c698489-2952v\" (UID: \"351ba54a-209f-4ec6-8f6a-56bbef8f3e68\") " pod="openshift-monitoring/metrics-server-7f4c698489-2952v"
Jul 17 10:59:38.574482 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.574304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/351ba54a-209f-4ec6-8f6a-56bbef8f3e68-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f4c698489-2952v\" (UID: \"351ba54a-209f-4ec6-8f6a-56bbef8f3e68\") " pod="openshift-monitoring/metrics-server-7f4c698489-2952v"
Jul 17 10:59:38.574482 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.574343    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/351ba54a-209f-4ec6-8f6a-56bbef8f3e68-secret-metrics-server-client-certs\") pod \"metrics-server-7f4c698489-2952v\" (UID: \"351ba54a-209f-4ec6-8f6a-56bbef8f3e68\") " pod="openshift-monitoring/metrics-server-7f4c698489-2952v"
Jul 17 10:59:38.574482 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.574361    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/351ba54a-209f-4ec6-8f6a-56bbef8f3e68-audit-log\") pod \"metrics-server-7f4c698489-2952v\" (UID: \"351ba54a-209f-4ec6-8f6a-56bbef8f3e68\") " pod="openshift-monitoring/metrics-server-7f4c698489-2952v"
Jul 17 10:59:38.574482 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.574383    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/351ba54a-209f-4ec6-8f6a-56bbef8f3e68-metrics-server-audit-profiles\") pod \"metrics-server-7f4c698489-2952v\" (UID: \"351ba54a-209f-4ec6-8f6a-56bbef8f3e68\") " pod="openshift-monitoring/metrics-server-7f4c698489-2952v"
Jul 17 10:59:38.574482 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.574438    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r88fs\" (UniqueName: \"kubernetes.io/projected/351ba54a-209f-4ec6-8f6a-56bbef8f3e68-kube-api-access-r88fs\") pod \"metrics-server-7f4c698489-2952v\" (UID: \"351ba54a-209f-4ec6-8f6a-56bbef8f3e68\") " pod="openshift-monitoring/metrics-server-7f4c698489-2952v"
Jul 17 10:59:38.675816 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.675777    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/351ba54a-209f-4ec6-8f6a-56bbef8f3e68-secret-metrics-server-client-certs\") pod \"metrics-server-7f4c698489-2952v\" (UID: \"351ba54a-209f-4ec6-8f6a-56bbef8f3e68\") " pod="openshift-monitoring/metrics-server-7f4c698489-2952v"
Jul 17 10:59:38.675978 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.675828    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/351ba54a-209f-4ec6-8f6a-56bbef8f3e68-audit-log\") pod \"metrics-server-7f4c698489-2952v\" (UID: \"351ba54a-209f-4ec6-8f6a-56bbef8f3e68\") " pod="openshift-monitoring/metrics-server-7f4c698489-2952v"
Jul 17 10:59:38.675978 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.675859    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/351ba54a-209f-4ec6-8f6a-56bbef8f3e68-metrics-server-audit-profiles\") pod \"metrics-server-7f4c698489-2952v\" (UID: \"351ba54a-209f-4ec6-8f6a-56bbef8f3e68\") " pod="openshift-monitoring/metrics-server-7f4c698489-2952v"
Jul 17 10:59:38.675978 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.675892    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r88fs\" (UniqueName: \"kubernetes.io/projected/351ba54a-209f-4ec6-8f6a-56bbef8f3e68-kube-api-access-r88fs\") pod \"metrics-server-7f4c698489-2952v\" (UID: \"351ba54a-209f-4ec6-8f6a-56bbef8f3e68\") " pod="openshift-monitoring/metrics-server-7f4c698489-2952v"
Jul 17 10:59:38.675978 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.675952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/351ba54a-209f-4ec6-8f6a-56bbef8f3e68-client-ca-bundle\") pod \"metrics-server-7f4c698489-2952v\" (UID: \"351ba54a-209f-4ec6-8f6a-56bbef8f3e68\") " pod="openshift-monitoring/metrics-server-7f4c698489-2952v"
Jul 17 10:59:38.676215 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.676005    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/351ba54a-209f-4ec6-8f6a-56bbef8f3e68-secret-metrics-server-tls\") pod \"metrics-server-7f4c698489-2952v\" (UID: \"351ba54a-209f-4ec6-8f6a-56bbef8f3e68\") " pod="openshift-monitoring/metrics-server-7f4c698489-2952v"
Jul 17 10:59:38.676215 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.676031    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/351ba54a-209f-4ec6-8f6a-56bbef8f3e68-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f4c698489-2952v\" (UID: \"351ba54a-209f-4ec6-8f6a-56bbef8f3e68\") " pod="openshift-monitoring/metrics-server-7f4c698489-2952v"
Jul 17 10:59:38.676885 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.676857    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/351ba54a-209f-4ec6-8f6a-56bbef8f3e68-audit-log\") pod \"metrics-server-7f4c698489-2952v\" (UID: \"351ba54a-209f-4ec6-8f6a-56bbef8f3e68\") " pod="openshift-monitoring/metrics-server-7f4c698489-2952v"
Jul 17 10:59:38.677104 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.677026    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/351ba54a-209f-4ec6-8f6a-56bbef8f3e68-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f4c698489-2952v\" (UID: \"351ba54a-209f-4ec6-8f6a-56bbef8f3e68\") " pod="openshift-monitoring/metrics-server-7f4c698489-2952v"
Jul 17 10:59:38.677372 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.677346    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/351ba54a-209f-4ec6-8f6a-56bbef8f3e68-metrics-server-audit-profiles\") pod \"metrics-server-7f4c698489-2952v\" (UID: \"351ba54a-209f-4ec6-8f6a-56bbef8f3e68\") " pod="openshift-monitoring/metrics-server-7f4c698489-2952v"
Jul 17 10:59:38.678706 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.678686    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/351ba54a-209f-4ec6-8f6a-56bbef8f3e68-secret-metrics-server-client-certs\") pod \"metrics-server-7f4c698489-2952v\" (UID: \"351ba54a-209f-4ec6-8f6a-56bbef8f3e68\") " pod="openshift-monitoring/metrics-server-7f4c698489-2952v"
Jul 17 10:59:38.678821 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.678800    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/351ba54a-209f-4ec6-8f6a-56bbef8f3e68-secret-metrics-server-tls\") pod \"metrics-server-7f4c698489-2952v\" (UID: \"351ba54a-209f-4ec6-8f6a-56bbef8f3e68\") " pod="openshift-monitoring/metrics-server-7f4c698489-2952v"
Jul 17 10:59:38.678957 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.678935    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/351ba54a-209f-4ec6-8f6a-56bbef8f3e68-client-ca-bundle\") pod \"metrics-server-7f4c698489-2952v\" (UID: \"351ba54a-209f-4ec6-8f6a-56bbef8f3e68\") " pod="openshift-monitoring/metrics-server-7f4c698489-2952v"
Jul 17 10:59:38.687421 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.687397    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r88fs\" (UniqueName: \"kubernetes.io/projected/351ba54a-209f-4ec6-8f6a-56bbef8f3e68-kube-api-access-r88fs\") pod \"metrics-server-7f4c698489-2952v\" (UID: \"351ba54a-209f-4ec6-8f6a-56bbef8f3e68\") " pod="openshift-monitoring/metrics-server-7f4c698489-2952v"
Jul 17 10:59:38.849714 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.849635    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f4c698489-2952v"
Jul 17 10:59:38.979855 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:38.979812    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:39.002424 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.002394    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f4c698489-2952v"]
Jul 17 10:59:39.005049 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:59:39.005028    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351ba54a_209f_4ec6_8f6a_56bbef8f3e68.slice/crio-222c0e3d6b04a2cbd125f6d3e7687d1a445a9747b60e77ad6d937a6b1bae2cd0 WatchSource:0}: Error finding container 222c0e3d6b04a2cbd125f6d3e7687d1a445a9747b60e77ad6d937a6b1bae2cd0: Status 404 returned error can't find the container with id 222c0e3d6b04a2cbd125f6d3e7687d1a445a9747b60e77ad6d937a6b1bae2cd0
Jul 17 10:59:39.081213 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.081184    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:39.269254 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.269230    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b67c7549b-8c4bq"]
Jul 17 10:59:39.301589 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.301550    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f4c698489-2952v" event={"ID":"351ba54a-209f-4ec6-8f6a-56bbef8f3e68","Type":"ContainerStarted","Data":"222c0e3d6b04a2cbd125f6d3e7687d1a445a9747b60e77ad6d937a6b1bae2cd0"}
Jul 17 10:59:39.303963 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.303931    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s" event={"ID":"f440b569-a68f-48e6-8a50-51b48d4e71e1","Type":"ContainerStarted","Data":"dfc913ba4b84cdaebfb9fd64af7d92af1f4c7f255f2357f6b9d23901456e4879"}
Jul 17 10:59:39.304058 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.303995    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s" event={"ID":"f440b569-a68f-48e6-8a50-51b48d4e71e1","Type":"ContainerStarted","Data":"d7e5fa4ff57243ca30933a19b6cc75e8e9fffed3d6392b2949f936e5cbc76823"}
Jul 17 10:59:39.304058 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.304023    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s" event={"ID":"f440b569-a68f-48e6-8a50-51b48d4e71e1","Type":"ContainerStarted","Data":"3020c6bd4519c6cbbc04dd3f9ce0f7ad87c85d75ae99ec8def72714093256b7d"}
Jul 17 10:59:39.315612 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.315584    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bbb6965f8-fgcpw"]
Jul 17 10:59:39.341603 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.341569    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bbb6965f8-fgcpw"]
Jul 17 10:59:39.341746 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.341687    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bbb6965f8-fgcpw"
Jul 17 10:59:39.349609 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.349584    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 17 10:59:39.381370 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.381296    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ad25c77-e6a7-4c3e-ae4d-06428457a784-console-config\") pod \"console-7bbb6965f8-fgcpw\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") " pod="openshift-console/console-7bbb6965f8-fgcpw"
Jul 17 10:59:39.381370 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.381346    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad25c77-e6a7-4c3e-ae4d-06428457a784-service-ca\") pod \"console-7bbb6965f8-fgcpw\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") " pod="openshift-console/console-7bbb6965f8-fgcpw"
Jul 17 10:59:39.381599 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.381383    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ad25c77-e6a7-4c3e-ae4d-06428457a784-console-oauth-config\") pod \"console-7bbb6965f8-fgcpw\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") " pod="openshift-console/console-7bbb6965f8-fgcpw"
Jul 17 10:59:39.381599 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.381439    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ad25c77-e6a7-4c3e-ae4d-06428457a784-trusted-ca-bundle\") pod \"console-7bbb6965f8-fgcpw\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") " pod="openshift-console/console-7bbb6965f8-fgcpw"
Jul 17 10:59:39.381599 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.381492    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ad25c77-e6a7-4c3e-ae4d-06428457a784-console-serving-cert\") pod \"console-7bbb6965f8-fgcpw\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") " pod="openshift-console/console-7bbb6965f8-fgcpw"
Jul 17 10:59:39.381599 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.381549    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ad25c77-e6a7-4c3e-ae4d-06428457a784-oauth-serving-cert\") pod \"console-7bbb6965f8-fgcpw\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") " pod="openshift-console/console-7bbb6965f8-fgcpw"
Jul 17 10:59:39.381763 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.381611    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hd29\" (UniqueName: \"kubernetes.io/projected/2ad25c77-e6a7-4c3e-ae4d-06428457a784-kube-api-access-4hd29\") pod \"console-7bbb6965f8-fgcpw\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") " pod="openshift-console/console-7bbb6965f8-fgcpw"
Jul 17 10:59:39.482774 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.482734    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ad25c77-e6a7-4c3e-ae4d-06428457a784-console-serving-cert\") pod \"console-7bbb6965f8-fgcpw\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") " pod="openshift-console/console-7bbb6965f8-fgcpw"
Jul 17 10:59:39.482958 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.482786    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ad25c77-e6a7-4c3e-ae4d-06428457a784-oauth-serving-cert\") pod \"console-7bbb6965f8-fgcpw\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") " pod="openshift-console/console-7bbb6965f8-fgcpw"
Jul 17 10:59:39.482958 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.482803    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hd29\" (UniqueName: \"kubernetes.io/projected/2ad25c77-e6a7-4c3e-ae4d-06428457a784-kube-api-access-4hd29\") pod \"console-7bbb6965f8-fgcpw\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") " pod="openshift-console/console-7bbb6965f8-fgcpw"
Jul 17 10:59:39.482958 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.482835    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ad25c77-e6a7-4c3e-ae4d-06428457a784-console-config\") pod \"console-7bbb6965f8-fgcpw\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") " pod="openshift-console/console-7bbb6965f8-fgcpw"
Jul 17 10:59:39.482958 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.482859    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad25c77-e6a7-4c3e-ae4d-06428457a784-service-ca\") pod \"console-7bbb6965f8-fgcpw\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") " pod="openshift-console/console-7bbb6965f8-fgcpw"
Jul 17 10:59:39.482958 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.482881    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ad25c77-e6a7-4c3e-ae4d-06428457a784-console-oauth-config\") pod \"console-7bbb6965f8-fgcpw\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") " pod="openshift-console/console-7bbb6965f8-fgcpw"
Jul 17 10:59:39.482958 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.482898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ad25c77-e6a7-4c3e-ae4d-06428457a784-trusted-ca-bundle\") pod \"console-7bbb6965f8-fgcpw\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") " pod="openshift-console/console-7bbb6965f8-fgcpw"
Jul 17 10:59:39.483690 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.483662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ad25c77-e6a7-4c3e-ae4d-06428457a784-oauth-serving-cert\") pod \"console-7bbb6965f8-fgcpw\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") " pod="openshift-console/console-7bbb6965f8-fgcpw"
Jul 17 10:59:39.483818 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.483662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad25c77-e6a7-4c3e-ae4d-06428457a784-service-ca\") pod \"console-7bbb6965f8-fgcpw\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") " pod="openshift-console/console-7bbb6965f8-fgcpw"
Jul 17 10:59:39.483877 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.483821    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ad25c77-e6a7-4c3e-ae4d-06428457a784-console-config\") pod \"console-7bbb6965f8-fgcpw\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") " pod="openshift-console/console-7bbb6965f8-fgcpw"
Jul 17 10:59:39.483978 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.483959    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ad25c77-e6a7-4c3e-ae4d-06428457a784-trusted-ca-bundle\") pod \"console-7bbb6965f8-fgcpw\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") " pod="openshift-console/console-7bbb6965f8-fgcpw"
Jul 17 10:59:39.485284 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.485265    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ad25c77-e6a7-4c3e-ae4d-06428457a784-console-serving-cert\") pod \"console-7bbb6965f8-fgcpw\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") " pod="openshift-console/console-7bbb6965f8-fgcpw"
Jul 17 10:59:39.485358 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.485336    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ad25c77-e6a7-4c3e-ae4d-06428457a784-console-oauth-config\") pod \"console-7bbb6965f8-fgcpw\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") " pod="openshift-console/console-7bbb6965f8-fgcpw"
Jul 17 10:59:39.494681 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.494657    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hd29\" (UniqueName: \"kubernetes.io/projected/2ad25c77-e6a7-4c3e-ae4d-06428457a784-kube-api-access-4hd29\") pod \"console-7bbb6965f8-fgcpw\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") " pod="openshift-console/console-7bbb6965f8-fgcpw"
Jul 17 10:59:39.541463 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.541430    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:39.606666 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.606606    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:39.650878 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.650785    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bbb6965f8-fgcpw"
Jul 17 10:59:39.814555 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:39.814519    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bbb6965f8-fgcpw"]
Jul 17 10:59:39.817427 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:59:39.817395    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad25c77_e6a7_4c3e_ae4d_06428457a784.slice/crio-bd2458e3751095277e081fe14e2a828786d2c24d3f514225191ecfc32c101bcd WatchSource:0}: Error finding container bd2458e3751095277e081fe14e2a828786d2c24d3f514225191ecfc32c101bcd: Status 404 returned error can't find the container with id bd2458e3751095277e081fe14e2a828786d2c24d3f514225191ecfc32c101bcd
Jul 17 10:59:40.313676 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:40.313592    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bbb6965f8-fgcpw" event={"ID":"2ad25c77-e6a7-4c3e-ae4d-06428457a784","Type":"ContainerStarted","Data":"3a2975d2a0e7400780edd6e2757d9f881cd4e4d7c73f51455e7593dbde205fdc"}
Jul 17 10:59:40.314263 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:40.313686    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bbb6965f8-fgcpw" event={"ID":"2ad25c77-e6a7-4c3e-ae4d-06428457a784","Type":"ContainerStarted","Data":"bd2458e3751095277e081fe14e2a828786d2c24d3f514225191ecfc32c101bcd"}
Jul 17 10:59:40.337142 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:40.337063    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bbb6965f8-fgcpw" podStartSLOduration=1.337044228 podStartE2EDuration="1.337044228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:59:40.334756869 +0000 UTC m=+135.264296510" watchObservedRunningTime="2026-07-17 10:59:40.337044228 +0000 UTC m=+135.266583876"
Jul 17 10:59:40.450048 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:40.449998    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:40.551992 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:40.551955    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:40.653290 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:40.653261    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:41.319656 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:41.319626    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f4c698489-2952v" event={"ID":"351ba54a-209f-4ec6-8f6a-56bbef8f3e68","Type":"ContainerStarted","Data":"12159693f950441af778bc59ba9e9a619b983cec86815e060bdcfd38fbc11c68"}
Jul 17 10:59:41.322556 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:41.322517    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s" event={"ID":"f440b569-a68f-48e6-8a50-51b48d4e71e1","Type":"ContainerStarted","Data":"20a8a286449748467b79619060d7a1432446a6e9216664f1a5773d2079233a9a"}
Jul 17 10:59:41.340763 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:41.340713    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7f4c698489-2952v" podStartSLOduration=1.214115614 podStartE2EDuration="3.340693901s" totalImagesPullingTime="2.126578287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:59:41.339712654 +0000 UTC m=+136.269252293" watchObservedRunningTime="2026-07-17 10:59:41.340693901 +0000 UTC m=+136.270233542"
Jul 17 10:59:42.329686 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:42.329649    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s" event={"ID":"f440b569-a68f-48e6-8a50-51b48d4e71e1","Type":"ContainerStarted","Data":"642e3ba7ebfccd6fcb129302b248987a98aec333fea6ce0ef2e681648ef3ff92"}
Jul 17 10:59:42.330118 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:42.329725    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s" event={"ID":"f440b569-a68f-48e6-8a50-51b48d4e71e1","Type":"ContainerStarted","Data":"9ca629a65430ec5a01befead4b7e0ac2684bece9d0efcd45f97baeac4f7b0930"}
Jul 17 10:59:42.356087 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:42.356027    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s" podStartSLOduration=2.307246213 podStartE2EDuration="6.356011234s" totalImagesPullingTime="4.048765021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:59:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:59:42.354611031 +0000 UTC m=+137.284150685" watchObservedRunningTime="2026-07-17 10:59:42.356011234 +0000 UTC m=+137.285550878"
Jul 17 10:59:43.333757 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:43.333724    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:44.096595 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.096563    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bbb6965f8-fgcpw"]
Jul 17 10:59:44.135423 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.135392    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86cccb7884-7ggcx"]
Jul 17 10:59:44.149863 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.149834    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 10:59:44.150032 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.149914    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86cccb7884-7ggcx"]
Jul 17 10:59:44.222669 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.222635    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4715d837-b3a3-4a2b-977d-b37e884f51ec-console-serving-cert\") pod \"console-86cccb7884-7ggcx\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") " pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 10:59:44.222857 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.222681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4715d837-b3a3-4a2b-977d-b37e884f51ec-service-ca\") pod \"console-86cccb7884-7ggcx\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") " pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 10:59:44.222857 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.222703    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ptzx\" (UniqueName: \"kubernetes.io/projected/4715d837-b3a3-4a2b-977d-b37e884f51ec-kube-api-access-8ptzx\") pod \"console-86cccb7884-7ggcx\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") " pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 10:59:44.222857 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.222765    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4715d837-b3a3-4a2b-977d-b37e884f51ec-trusted-ca-bundle\") pod \"console-86cccb7884-7ggcx\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") " pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 10:59:44.222857 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.222828    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4715d837-b3a3-4a2b-977d-b37e884f51ec-console-config\") pod \"console-86cccb7884-7ggcx\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") " pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 10:59:44.222857 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.222851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4715d837-b3a3-4a2b-977d-b37e884f51ec-oauth-serving-cert\") pod \"console-86cccb7884-7ggcx\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") " pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 10:59:44.223019 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.222923    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4715d837-b3a3-4a2b-977d-b37e884f51ec-console-oauth-config\") pod \"console-86cccb7884-7ggcx\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") " pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 10:59:44.295422 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.295391    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:44.323920 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.323881    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ptzx\" (UniqueName: \"kubernetes.io/projected/4715d837-b3a3-4a2b-977d-b37e884f51ec-kube-api-access-8ptzx\") pod \"console-86cccb7884-7ggcx\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") " pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 10:59:44.323920 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.323925    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4715d837-b3a3-4a2b-977d-b37e884f51ec-trusted-ca-bundle\") pod \"console-86cccb7884-7ggcx\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") " pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 10:59:44.324222 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.323971    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4715d837-b3a3-4a2b-977d-b37e884f51ec-console-config\") pod \"console-86cccb7884-7ggcx\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") " pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 10:59:44.324222 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.323987    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4715d837-b3a3-4a2b-977d-b37e884f51ec-oauth-serving-cert\") pod \"console-86cccb7884-7ggcx\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") " pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 10:59:44.324222 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.324041    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4715d837-b3a3-4a2b-977d-b37e884f51ec-console-oauth-config\") pod \"console-86cccb7884-7ggcx\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") " pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 10:59:44.324222 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.324108    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4715d837-b3a3-4a2b-977d-b37e884f51ec-console-serving-cert\") pod \"console-86cccb7884-7ggcx\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") " pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 10:59:44.324222 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.324144    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4715d837-b3a3-4a2b-977d-b37e884f51ec-service-ca\") pod \"console-86cccb7884-7ggcx\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") " pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 10:59:44.324775 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.324752    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4715d837-b3a3-4a2b-977d-b37e884f51ec-console-config\") pod \"console-86cccb7884-7ggcx\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") " pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 10:59:44.324882 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.324857    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4715d837-b3a3-4a2b-977d-b37e884f51ec-oauth-serving-cert\") pod \"console-86cccb7884-7ggcx\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") " pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 10:59:44.324954 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.324936    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4715d837-b3a3-4a2b-977d-b37e884f51ec-trusted-ca-bundle\") pod \"console-86cccb7884-7ggcx\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") " pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 10:59:44.324990 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.324943    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4715d837-b3a3-4a2b-977d-b37e884f51ec-service-ca\") pod \"console-86cccb7884-7ggcx\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") " pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 10:59:44.326546 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.326526    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4715d837-b3a3-4a2b-977d-b37e884f51ec-console-serving-cert\") pod \"console-86cccb7884-7ggcx\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") " pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 10:59:44.326631 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.326548    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4715d837-b3a3-4a2b-977d-b37e884f51ec-console-oauth-config\") pod \"console-86cccb7884-7ggcx\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") " pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 10:59:44.338104 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.336030    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ptzx\" (UniqueName: \"kubernetes.io/projected/4715d837-b3a3-4a2b-977d-b37e884f51ec-kube-api-access-8ptzx\") pod \"console-86cccb7884-7ggcx\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") " pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 10:59:44.345598 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.345579    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-78cd79d867-5lv8s"
Jul 17 10:59:44.362488 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.362420    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:44.459061 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.459024    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 10:59:44.592112 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.592060    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86cccb7884-7ggcx"]
Jul 17 10:59:44.595027 ip-10-0-140-233 kubenswrapper[2563]: W0717 10:59:44.595000    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4715d837_b3a3_4a2b_977d_b37e884f51ec.slice/crio-69ed61e5418c9eaa13846890ae267d9290f6abe05e35ed8e75477aadc6b5be81 WatchSource:0}: Error finding container 69ed61e5418c9eaa13846890ae267d9290f6abe05e35ed8e75477aadc6b5be81: Status 404 returned error can't find the container with id 69ed61e5418c9eaa13846890ae267d9290f6abe05e35ed8e75477aadc6b5be81
Jul 17 10:59:44.890216 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.890188    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:44.992035 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:44.992008    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:45.345340 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:45.345254    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86cccb7884-7ggcx" event={"ID":"4715d837-b3a3-4a2b-977d-b37e884f51ec","Type":"ContainerStarted","Data":"61ed74119cd5ef02c0f51dadc9563e272ca2db804ccd8e23754387a0b4fd1274"}
Jul 17 10:59:45.345340 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:45.345306    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86cccb7884-7ggcx" event={"ID":"4715d837-b3a3-4a2b-977d-b37e884f51ec","Type":"ContainerStarted","Data":"69ed61e5418c9eaa13846890ae267d9290f6abe05e35ed8e75477aadc6b5be81"}
Jul 17 10:59:45.365200 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:45.365140    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86cccb7884-7ggcx" podStartSLOduration=1.36512371 podStartE2EDuration="1.36512371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:59:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:59:45.364021099 +0000 UTC m=+140.293560737" watchObservedRunningTime="2026-07-17 10:59:45.36512371 +0000 UTC m=+140.294663351"
Jul 17 10:59:49.651509 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:49.651469    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bbb6965f8-fgcpw"
Jul 17 10:59:50.995153 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:50.995121    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:51.096651 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:51.096615    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:54.459515 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:54.459470    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 10:59:54.459977 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:54.459622    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 10:59:54.464490 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:54.464461    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 10:59:55.390700 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:55.390671    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 10:59:55.584694 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:55.584662    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:55.685050 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:55.684970    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:55.752318 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:55.752292    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 10:59:56.391813 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:56.391775    2563 generic.go:356] "Generic (PLEG): container finished" podID="7cd8f40f-2f2d-47e1-b464-33828197574b" containerID="3bc19463bc32b4fc02b4cd51690290469753d9ce5a6724d67dc9a43ec153d847" exitCode=0
Jul 17 10:59:56.392032 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:56.391870    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-h94fh" event={"ID":"7cd8f40f-2f2d-47e1-b464-33828197574b","Type":"ContainerDied","Data":"3bc19463bc32b4fc02b4cd51690290469753d9ce5a6724d67dc9a43ec153d847"}
Jul 17 10:59:56.392349 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:56.392332    2563 scope.go:117] "RemoveContainer" containerID="3bc19463bc32b4fc02b4cd51690290469753d9ce5a6724d67dc9a43ec153d847"
Jul 17 10:59:57.397206 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:57.397165    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-h94fh" event={"ID":"7cd8f40f-2f2d-47e1-b464-33828197574b","Type":"ContainerStarted","Data":"a9df72f27e7564f4584364f3a1f368145b31d1f29e24d3a9d27a06c85047a760"}
Jul 17 10:59:58.850122 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:58.850088    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7f4c698489-2952v"
Jul 17 10:59:58.850521 ip-10-0-140-233 kubenswrapper[2563]: I0717 10:59:58.850144    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7f4c698489-2952v"
Jul 17 11:00:04.289718 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:04.289656    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b67c7549b-8c4bq" podUID="5d4261da-e6d0-40f1-a212-606204bdd893" containerName="console" containerID="cri-o://d3f835a9e2200c2f9756bb935657e70b402c5f7436260ba37aded999b57c230b" gracePeriod=15
Jul 17 11:00:04.414061 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:04.414033    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:00:04.426037 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:04.426001    2563 generic.go:356] "Generic (PLEG): container finished" podID="5d4261da-e6d0-40f1-a212-606204bdd893" containerID="d3f835a9e2200c2f9756bb935657e70b402c5f7436260ba37aded999b57c230b" exitCode=0
Jul 17 11:00:04.426192 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:04.426091    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b67c7549b-8c4bq" event={"ID":"5d4261da-e6d0-40f1-a212-606204bdd893","Type":"ContainerDied","Data":"d3f835a9e2200c2f9756bb935657e70b402c5f7436260ba37aded999b57c230b"}
Jul 17 11:00:04.477404 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:04.477316    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:00:04.531207 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:04.531184    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b67c7549b-8c4bq"
Jul 17 11:00:04.711470 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:04.711427    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d4261da-e6d0-40f1-a212-606204bdd893-service-ca\") pod \"5d4261da-e6d0-40f1-a212-606204bdd893\" (UID: \"5d4261da-e6d0-40f1-a212-606204bdd893\") "
Jul 17 11:00:04.711658 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:04.711486    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6mzw\" (UniqueName: \"kubernetes.io/projected/5d4261da-e6d0-40f1-a212-606204bdd893-kube-api-access-f6mzw\") pod \"5d4261da-e6d0-40f1-a212-606204bdd893\" (UID: \"5d4261da-e6d0-40f1-a212-606204bdd893\") "
Jul 17 11:00:04.711658 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:04.711527    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d4261da-e6d0-40f1-a212-606204bdd893-console-config\") pod \"5d4261da-e6d0-40f1-a212-606204bdd893\" (UID: \"5d4261da-e6d0-40f1-a212-606204bdd893\") "
Jul 17 11:00:04.711658 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:04.711551    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d4261da-e6d0-40f1-a212-606204bdd893-console-serving-cert\") pod \"5d4261da-e6d0-40f1-a212-606204bdd893\" (UID: \"5d4261da-e6d0-40f1-a212-606204bdd893\") "
Jul 17 11:00:04.711658 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:04.711571    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d4261da-e6d0-40f1-a212-606204bdd893-console-oauth-config\") pod \"5d4261da-e6d0-40f1-a212-606204bdd893\" (UID: \"5d4261da-e6d0-40f1-a212-606204bdd893\") "
Jul 17 11:00:04.711658 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:04.711623    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d4261da-e6d0-40f1-a212-606204bdd893-oauth-serving-cert\") pod \"5d4261da-e6d0-40f1-a212-606204bdd893\" (UID: \"5d4261da-e6d0-40f1-a212-606204bdd893\") "
Jul 17 11:00:04.711997 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:04.711964    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d4261da-e6d0-40f1-a212-606204bdd893-service-ca" (OuterVolumeSpecName: "service-ca") pod "5d4261da-e6d0-40f1-a212-606204bdd893" (UID: "5d4261da-e6d0-40f1-a212-606204bdd893"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:00:04.712123 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:04.711968    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d4261da-e6d0-40f1-a212-606204bdd893-console-config" (OuterVolumeSpecName: "console-config") pod "5d4261da-e6d0-40f1-a212-606204bdd893" (UID: "5d4261da-e6d0-40f1-a212-606204bdd893"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:00:04.712123 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:04.712018    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d4261da-e6d0-40f1-a212-606204bdd893-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5d4261da-e6d0-40f1-a212-606204bdd893" (UID: "5d4261da-e6d0-40f1-a212-606204bdd893"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:00:04.713787 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:04.713764    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d4261da-e6d0-40f1-a212-606204bdd893-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5d4261da-e6d0-40f1-a212-606204bdd893" (UID: "5d4261da-e6d0-40f1-a212-606204bdd893"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 11:00:04.713875 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:04.713813    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d4261da-e6d0-40f1-a212-606204bdd893-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5d4261da-e6d0-40f1-a212-606204bdd893" (UID: "5d4261da-e6d0-40f1-a212-606204bdd893"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 11:00:04.713954 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:04.713870    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d4261da-e6d0-40f1-a212-606204bdd893-kube-api-access-f6mzw" (OuterVolumeSpecName: "kube-api-access-f6mzw") pod "5d4261da-e6d0-40f1-a212-606204bdd893" (UID: "5d4261da-e6d0-40f1-a212-606204bdd893"). InnerVolumeSpecName "kube-api-access-f6mzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 11:00:04.812466 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:04.812426    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d4261da-e6d0-40f1-a212-606204bdd893-oauth-serving-cert\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:00:04.812466 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:04.812455    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d4261da-e6d0-40f1-a212-606204bdd893-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:00:04.812466 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:04.812466    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6mzw\" (UniqueName: \"kubernetes.io/projected/5d4261da-e6d0-40f1-a212-606204bdd893-kube-api-access-f6mzw\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:00:04.812693 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:04.812480    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d4261da-e6d0-40f1-a212-606204bdd893-console-config\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:00:04.812693 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:04.812494    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d4261da-e6d0-40f1-a212-606204bdd893-console-serving-cert\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:00:04.812693 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:04.812505    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d4261da-e6d0-40f1-a212-606204bdd893-console-oauth-config\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:00:05.431522 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:05.431488    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b67c7549b-8c4bq" event={"ID":"5d4261da-e6d0-40f1-a212-606204bdd893","Type":"ContainerDied","Data":"e2e9ae1a2195f05bef34e0190fef58fe9324dd9c32585226ec40f4adbd722079"}
Jul 17 11:00:05.431986 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:05.431527    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b67c7549b-8c4bq"
Jul 17 11:00:05.431986 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:05.431546    2563 scope.go:117] "RemoveContainer" containerID="d3f835a9e2200c2f9756bb935657e70b402c5f7436260ba37aded999b57c230b"
Jul 17 11:00:05.454026 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:05.453875    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b67c7549b-8c4bq"]
Jul 17 11:00:05.460747 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:05.460725    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b67c7549b-8c4bq"]
Jul 17 11:00:05.664808 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:05.664772    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d4261da-e6d0-40f1-a212-606204bdd893" path="/var/lib/kubelet/pods/5d4261da-e6d0-40f1-a212-606204bdd893/volumes"
Jul 17 11:00:09.116434 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.116385    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bbb6965f8-fgcpw" podUID="2ad25c77-e6a7-4c3e-ae4d-06428457a784" containerName="console" containerID="cri-o://3a2975d2a0e7400780edd6e2757d9f881cd4e4d7c73f51455e7593dbde205fdc" gracePeriod=15
Jul 17 11:00:09.375767 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.375700    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bbb6965f8-fgcpw"
Jul 17 11:00:09.447675 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.447644    2563 generic.go:356] "Generic (PLEG): container finished" podID="2ad25c77-e6a7-4c3e-ae4d-06428457a784" containerID="3a2975d2a0e7400780edd6e2757d9f881cd4e4d7c73f51455e7593dbde205fdc" exitCode=0
Jul 17 11:00:09.447840 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.447690    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bbb6965f8-fgcpw"
Jul 17 11:00:09.447840 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.447730    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bbb6965f8-fgcpw" event={"ID":"2ad25c77-e6a7-4c3e-ae4d-06428457a784","Type":"ContainerDied","Data":"3a2975d2a0e7400780edd6e2757d9f881cd4e4d7c73f51455e7593dbde205fdc"}
Jul 17 11:00:09.447840 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.447779    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bbb6965f8-fgcpw" event={"ID":"2ad25c77-e6a7-4c3e-ae4d-06428457a784","Type":"ContainerDied","Data":"bd2458e3751095277e081fe14e2a828786d2c24d3f514225191ecfc32c101bcd"}
Jul 17 11:00:09.447840 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.447801    2563 scope.go:117] "RemoveContainer" containerID="3a2975d2a0e7400780edd6e2757d9f881cd4e4d7c73f51455e7593dbde205fdc"
Jul 17 11:00:09.459793 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.459767    2563 scope.go:117] "RemoveContainer" containerID="3a2975d2a0e7400780edd6e2757d9f881cd4e4d7c73f51455e7593dbde205fdc"
Jul 17 11:00:09.460038 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:00:09.460017    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a2975d2a0e7400780edd6e2757d9f881cd4e4d7c73f51455e7593dbde205fdc\": container with ID starting with 3a2975d2a0e7400780edd6e2757d9f881cd4e4d7c73f51455e7593dbde205fdc not found: ID does not exist" containerID="3a2975d2a0e7400780edd6e2757d9f881cd4e4d7c73f51455e7593dbde205fdc"
Jul 17 11:00:09.460105 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.460045    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a2975d2a0e7400780edd6e2757d9f881cd4e4d7c73f51455e7593dbde205fdc"} err="failed to get container status \"3a2975d2a0e7400780edd6e2757d9f881cd4e4d7c73f51455e7593dbde205fdc\": rpc error: code = NotFound desc = could not find container \"3a2975d2a0e7400780edd6e2757d9f881cd4e4d7c73f51455e7593dbde205fdc\": container with ID starting with 3a2975d2a0e7400780edd6e2757d9f881cd4e4d7c73f51455e7593dbde205fdc not found: ID does not exist"
Jul 17 11:00:09.557024 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.556984    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ad25c77-e6a7-4c3e-ae4d-06428457a784-console-config\") pod \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") "
Jul 17 11:00:09.557024 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.557027    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hd29\" (UniqueName: \"kubernetes.io/projected/2ad25c77-e6a7-4c3e-ae4d-06428457a784-kube-api-access-4hd29\") pod \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") "
Jul 17 11:00:09.557254 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.557059    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad25c77-e6a7-4c3e-ae4d-06428457a784-service-ca\") pod \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") "
Jul 17 11:00:09.557254 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.557104    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ad25c77-e6a7-4c3e-ae4d-06428457a784-trusted-ca-bundle\") pod \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") "
Jul 17 11:00:09.557254 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.557142    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ad25c77-e6a7-4c3e-ae4d-06428457a784-console-oauth-config\") pod \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") "
Jul 17 11:00:09.557254 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.557169    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ad25c77-e6a7-4c3e-ae4d-06428457a784-console-serving-cert\") pod \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") "
Jul 17 11:00:09.557254 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.557239    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ad25c77-e6a7-4c3e-ae4d-06428457a784-oauth-serving-cert\") pod \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\" (UID: \"2ad25c77-e6a7-4c3e-ae4d-06428457a784\") "
Jul 17 11:00:09.557501 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.557474    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad25c77-e6a7-4c3e-ae4d-06428457a784-service-ca" (OuterVolumeSpecName: "service-ca") pod "2ad25c77-e6a7-4c3e-ae4d-06428457a784" (UID: "2ad25c77-e6a7-4c3e-ae4d-06428457a784"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:00:09.557564 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.557470    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad25c77-e6a7-4c3e-ae4d-06428457a784-console-config" (OuterVolumeSpecName: "console-config") pod "2ad25c77-e6a7-4c3e-ae4d-06428457a784" (UID: "2ad25c77-e6a7-4c3e-ae4d-06428457a784"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:00:09.557564 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.557529    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad25c77-e6a7-4c3e-ae4d-06428457a784-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2ad25c77-e6a7-4c3e-ae4d-06428457a784" (UID: "2ad25c77-e6a7-4c3e-ae4d-06428457a784"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:00:09.557789 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.557766    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad25c77-e6a7-4c3e-ae4d-06428457a784-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2ad25c77-e6a7-4c3e-ae4d-06428457a784" (UID: "2ad25c77-e6a7-4c3e-ae4d-06428457a784"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:00:09.559345 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.559310    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ad25c77-e6a7-4c3e-ae4d-06428457a784-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2ad25c77-e6a7-4c3e-ae4d-06428457a784" (UID: "2ad25c77-e6a7-4c3e-ae4d-06428457a784"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 11:00:09.559345 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.559325    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ad25c77-e6a7-4c3e-ae4d-06428457a784-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2ad25c77-e6a7-4c3e-ae4d-06428457a784" (UID: "2ad25c77-e6a7-4c3e-ae4d-06428457a784"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 11:00:09.559345 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.559318    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad25c77-e6a7-4c3e-ae4d-06428457a784-kube-api-access-4hd29" (OuterVolumeSpecName: "kube-api-access-4hd29") pod "2ad25c77-e6a7-4c3e-ae4d-06428457a784" (UID: "2ad25c77-e6a7-4c3e-ae4d-06428457a784"). InnerVolumeSpecName "kube-api-access-4hd29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 11:00:09.658706 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.658672    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ad25c77-e6a7-4c3e-ae4d-06428457a784-oauth-serving-cert\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:00:09.658706 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.658707    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ad25c77-e6a7-4c3e-ae4d-06428457a784-console-config\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:00:09.658894 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.658717    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hd29\" (UniqueName: \"kubernetes.io/projected/2ad25c77-e6a7-4c3e-ae4d-06428457a784-kube-api-access-4hd29\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:00:09.658894 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.658726    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad25c77-e6a7-4c3e-ae4d-06428457a784-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:00:09.658894 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.658736    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ad25c77-e6a7-4c3e-ae4d-06428457a784-trusted-ca-bundle\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:00:09.658894 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.658745    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ad25c77-e6a7-4c3e-ae4d-06428457a784-console-oauth-config\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:00:09.658894 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.658753    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ad25c77-e6a7-4c3e-ae4d-06428457a784-console-serving-cert\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:00:09.764313 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.764283    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bbb6965f8-fgcpw"]
Jul 17 11:00:09.769245 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:09.769222    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bbb6965f8-fgcpw"]
Jul 17 11:00:11.664320 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:11.664284    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ad25c77-e6a7-4c3e-ae4d-06428457a784" path="/var/lib/kubelet/pods/2ad25c77-e6a7-4c3e-ae4d-06428457a784/volumes"
Jul 17 11:00:17.094646 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:17.094559    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:00:18.855131 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:18.855098    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7f4c698489-2952v"
Jul 17 11:00:18.858959 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:18.858930    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7f4c698489-2952v"
Jul 17 11:00:45.144902 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:45.144864    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:00:50.378393 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:50.378356    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:00:50.582272 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:00:50.582238    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:01:01.313377 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:01.313327    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:01:01.414956 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:01.414920    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:01:01.514186 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:01.514153    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:01:02.033577 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:02.033546    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:01:02.096608 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:02.096575    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:01:02.443837 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:02.443791    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:01:02.541156 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:02.541123    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:01:06.013907 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:06.013873    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:01:06.111200 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:06.111171    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:01:06.186501 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:06.186475    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:01:09.448605 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:09.448572    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:01:09.552278 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:09.552246    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:01:11.642573 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:11.642528    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86cccb7884-7ggcx"]
Jul 17 11:01:11.762913 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:11.762883    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:01:11.863566 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:11.863539    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:01:11.925097 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:11.925015    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:01:20.598653 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:20.598618    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:01:33.626048 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:33.626011    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:01:34.004671 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:34.004631    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:01:36.662549 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:36.662495    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-86cccb7884-7ggcx" podUID="4715d837-b3a3-4a2b-977d-b37e884f51ec" containerName="console" containerID="cri-o://61ed74119cd5ef02c0f51dadc9563e272ca2db804ccd8e23754387a0b4fd1274" gracePeriod=15
Jul 17 11:01:36.899383 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:36.899355    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 11:01:36.961735 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:36.961643    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4715d837-b3a3-4a2b-977d-b37e884f51ec-console-serving-cert\") pod \"4715d837-b3a3-4a2b-977d-b37e884f51ec\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") "
Jul 17 11:01:36.961735 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:36.961681    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4715d837-b3a3-4a2b-977d-b37e884f51ec-trusted-ca-bundle\") pod \"4715d837-b3a3-4a2b-977d-b37e884f51ec\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") "
Jul 17 11:01:36.961735 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:36.961700    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4715d837-b3a3-4a2b-977d-b37e884f51ec-console-oauth-config\") pod \"4715d837-b3a3-4a2b-977d-b37e884f51ec\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") "
Jul 17 11:01:36.961735 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:36.961727    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4715d837-b3a3-4a2b-977d-b37e884f51ec-oauth-serving-cert\") pod \"4715d837-b3a3-4a2b-977d-b37e884f51ec\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") "
Jul 17 11:01:36.962025 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:36.961774    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4715d837-b3a3-4a2b-977d-b37e884f51ec-service-ca\") pod \"4715d837-b3a3-4a2b-977d-b37e884f51ec\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") "
Jul 17 11:01:36.962025 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:36.961805    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ptzx\" (UniqueName: \"kubernetes.io/projected/4715d837-b3a3-4a2b-977d-b37e884f51ec-kube-api-access-8ptzx\") pod \"4715d837-b3a3-4a2b-977d-b37e884f51ec\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") "
Jul 17 11:01:36.962025 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:36.961936    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4715d837-b3a3-4a2b-977d-b37e884f51ec-console-config\") pod \"4715d837-b3a3-4a2b-977d-b37e884f51ec\" (UID: \"4715d837-b3a3-4a2b-977d-b37e884f51ec\") "
Jul 17 11:01:36.962210 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:36.962146    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4715d837-b3a3-4a2b-977d-b37e884f51ec-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4715d837-b3a3-4a2b-977d-b37e884f51ec" (UID: "4715d837-b3a3-4a2b-977d-b37e884f51ec"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:01:36.962341 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:36.962315    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4715d837-b3a3-4a2b-977d-b37e884f51ec-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4715d837-b3a3-4a2b-977d-b37e884f51ec" (UID: "4715d837-b3a3-4a2b-977d-b37e884f51ec"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:01:36.962402 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:36.962311    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4715d837-b3a3-4a2b-977d-b37e884f51ec-service-ca" (OuterVolumeSpecName: "service-ca") pod "4715d837-b3a3-4a2b-977d-b37e884f51ec" (UID: "4715d837-b3a3-4a2b-977d-b37e884f51ec"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:01:36.962402 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:36.962337    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4715d837-b3a3-4a2b-977d-b37e884f51ec-console-config" (OuterVolumeSpecName: "console-config") pod "4715d837-b3a3-4a2b-977d-b37e884f51ec" (UID: "4715d837-b3a3-4a2b-977d-b37e884f51ec"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:01:36.963906 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:36.963880    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4715d837-b3a3-4a2b-977d-b37e884f51ec-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4715d837-b3a3-4a2b-977d-b37e884f51ec" (UID: "4715d837-b3a3-4a2b-977d-b37e884f51ec"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 11:01:36.964024 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:36.964005    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4715d837-b3a3-4a2b-977d-b37e884f51ec-kube-api-access-8ptzx" (OuterVolumeSpecName: "kube-api-access-8ptzx") pod "4715d837-b3a3-4a2b-977d-b37e884f51ec" (UID: "4715d837-b3a3-4a2b-977d-b37e884f51ec"). InnerVolumeSpecName "kube-api-access-8ptzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 11:01:36.964083 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:36.964018    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4715d837-b3a3-4a2b-977d-b37e884f51ec-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4715d837-b3a3-4a2b-977d-b37e884f51ec" (UID: "4715d837-b3a3-4a2b-977d-b37e884f51ec"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 11:01:37.063260 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:37.063216    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4715d837-b3a3-4a2b-977d-b37e884f51ec-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:01:37.063260 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:37.063253    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ptzx\" (UniqueName: \"kubernetes.io/projected/4715d837-b3a3-4a2b-977d-b37e884f51ec-kube-api-access-8ptzx\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:01:37.063260 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:37.063263    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4715d837-b3a3-4a2b-977d-b37e884f51ec-console-config\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:01:37.063260 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:37.063273    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4715d837-b3a3-4a2b-977d-b37e884f51ec-console-serving-cert\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:01:37.063260 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:37.063294    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4715d837-b3a3-4a2b-977d-b37e884f51ec-trusted-ca-bundle\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:01:37.063556 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:37.063303    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4715d837-b3a3-4a2b-977d-b37e884f51ec-console-oauth-config\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:01:37.063556 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:37.063311    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4715d837-b3a3-4a2b-977d-b37e884f51ec-oauth-serving-cert\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:01:37.761492 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:37.761455    2563 generic.go:356] "Generic (PLEG): container finished" podID="4715d837-b3a3-4a2b-977d-b37e884f51ec" containerID="61ed74119cd5ef02c0f51dadc9563e272ca2db804ccd8e23754387a0b4fd1274" exitCode=0
Jul 17 11:01:37.761916 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:37.761500    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86cccb7884-7ggcx"
Jul 17 11:01:37.761916 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:37.761514    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86cccb7884-7ggcx" event={"ID":"4715d837-b3a3-4a2b-977d-b37e884f51ec","Type":"ContainerDied","Data":"61ed74119cd5ef02c0f51dadc9563e272ca2db804ccd8e23754387a0b4fd1274"}
Jul 17 11:01:37.761916 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:37.761560    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86cccb7884-7ggcx" event={"ID":"4715d837-b3a3-4a2b-977d-b37e884f51ec","Type":"ContainerDied","Data":"69ed61e5418c9eaa13846890ae267d9290f6abe05e35ed8e75477aadc6b5be81"}
Jul 17 11:01:37.761916 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:37.761581    2563 scope.go:117] "RemoveContainer" containerID="61ed74119cd5ef02c0f51dadc9563e272ca2db804ccd8e23754387a0b4fd1274"
Jul 17 11:01:37.769541 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:37.769524    2563 scope.go:117] "RemoveContainer" containerID="61ed74119cd5ef02c0f51dadc9563e272ca2db804ccd8e23754387a0b4fd1274"
Jul 17 11:01:37.769836 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:01:37.769818    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61ed74119cd5ef02c0f51dadc9563e272ca2db804ccd8e23754387a0b4fd1274\": container with ID starting with 61ed74119cd5ef02c0f51dadc9563e272ca2db804ccd8e23754387a0b4fd1274 not found: ID does not exist" containerID="61ed74119cd5ef02c0f51dadc9563e272ca2db804ccd8e23754387a0b4fd1274"
Jul 17 11:01:37.769896 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:37.769844    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61ed74119cd5ef02c0f51dadc9563e272ca2db804ccd8e23754387a0b4fd1274"} err="failed to get container status \"61ed74119cd5ef02c0f51dadc9563e272ca2db804ccd8e23754387a0b4fd1274\": rpc error: code = NotFound desc = could not find container \"61ed74119cd5ef02c0f51dadc9563e272ca2db804ccd8e23754387a0b4fd1274\": container with ID starting with 61ed74119cd5ef02c0f51dadc9563e272ca2db804ccd8e23754387a0b4fd1274 not found: ID does not exist"
Jul 17 11:01:37.781619 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:37.781588    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86cccb7884-7ggcx"]
Jul 17 11:01:37.788941 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:37.788922    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86cccb7884-7ggcx"]
Jul 17 11:01:39.664232 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:39.664190    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4715d837-b3a3-4a2b-977d-b37e884f51ec" path="/var/lib/kubelet/pods/4715d837-b3a3-4a2b-977d-b37e884f51ec/volumes"
Jul 17 11:01:43.774751 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:43.774721    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:01:43.913833 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:43.913798    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:01:44.721308 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:44.721271    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5"]
Jul 17 11:01:44.721603 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:44.721585    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ad25c77-e6a7-4c3e-ae4d-06428457a784" containerName="console"
Jul 17 11:01:44.721673 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:44.721605    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ad25c77-e6a7-4c3e-ae4d-06428457a784" containerName="console"
Jul 17 11:01:44.721673 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:44.721618    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4715d837-b3a3-4a2b-977d-b37e884f51ec" containerName="console"
Jul 17 11:01:44.721673 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:44.721625    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4715d837-b3a3-4a2b-977d-b37e884f51ec" containerName="console"
Jul 17 11:01:44.721673 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:44.721658    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d4261da-e6d0-40f1-a212-606204bdd893" containerName="console"
Jul 17 11:01:44.721673 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:44.721666    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d4261da-e6d0-40f1-a212-606204bdd893" containerName="console"
Jul 17 11:01:44.721908 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:44.721728    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4715d837-b3a3-4a2b-977d-b37e884f51ec" containerName="console"
Jul 17 11:01:44.721908 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:44.721742    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ad25c77-e6a7-4c3e-ae4d-06428457a784" containerName="console"
Jul 17 11:01:44.721908 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:44.721751    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d4261da-e6d0-40f1-a212-606204bdd893" containerName="console"
Jul 17 11:01:44.726177 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:44.726155    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5"
Jul 17 11:01:44.729047 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:44.729025    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 17 11:01:44.729233 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:44.729216    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 17 11:01:44.730047 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:44.730030    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gxg5n\""
Jul 17 11:01:44.739571 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:44.739548    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5"]
Jul 17 11:01:44.831142 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:44.831107    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c0bed99-7109-4dd2-87be-62d46d41a252-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5\" (UID: \"9c0bed99-7109-4dd2-87be-62d46d41a252\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5"
Jul 17 11:01:44.831530 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:44.831162    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mpdm\" (UniqueName: \"kubernetes.io/projected/9c0bed99-7109-4dd2-87be-62d46d41a252-kube-api-access-7mpdm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5\" (UID: \"9c0bed99-7109-4dd2-87be-62d46d41a252\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5"
Jul 17 11:01:44.831530 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:44.831231    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c0bed99-7109-4dd2-87be-62d46d41a252-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5\" (UID: \"9c0bed99-7109-4dd2-87be-62d46d41a252\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5"
Jul 17 11:01:44.932473 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:44.932433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c0bed99-7109-4dd2-87be-62d46d41a252-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5\" (UID: \"9c0bed99-7109-4dd2-87be-62d46d41a252\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5"
Jul 17 11:01:44.932656 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:44.932487    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mpdm\" (UniqueName: \"kubernetes.io/projected/9c0bed99-7109-4dd2-87be-62d46d41a252-kube-api-access-7mpdm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5\" (UID: \"9c0bed99-7109-4dd2-87be-62d46d41a252\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5"
Jul 17 11:01:44.932656 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:44.932513    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c0bed99-7109-4dd2-87be-62d46d41a252-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5\" (UID: \"9c0bed99-7109-4dd2-87be-62d46d41a252\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5"
Jul 17 11:01:44.932885 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:44.932867    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c0bed99-7109-4dd2-87be-62d46d41a252-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5\" (UID: \"9c0bed99-7109-4dd2-87be-62d46d41a252\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5"
Jul 17 11:01:44.932919 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:44.932849    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c0bed99-7109-4dd2-87be-62d46d41a252-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5\" (UID: \"9c0bed99-7109-4dd2-87be-62d46d41a252\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5"
Jul 17 11:01:44.942382 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:44.942358    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mpdm\" (UniqueName: \"kubernetes.io/projected/9c0bed99-7109-4dd2-87be-62d46d41a252-kube-api-access-7mpdm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5\" (UID: \"9c0bed99-7109-4dd2-87be-62d46d41a252\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5"
Jul 17 11:01:45.034578 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:45.034493    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5"
Jul 17 11:01:45.165624 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:45.165545    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5"]
Jul 17 11:01:45.168293 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:01:45.168266    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0bed99_7109_4dd2_87be_62d46d41a252.slice/crio-2b69d8fc899473f57cb5414e6effc811630494eeb99e70c59dc97867e208ee8f WatchSource:0}: Error finding container 2b69d8fc899473f57cb5414e6effc811630494eeb99e70c59dc97867e208ee8f: Status 404 returned error can't find the container with id 2b69d8fc899473f57cb5414e6effc811630494eeb99e70c59dc97867e208ee8f
Jul 17 11:01:45.796315 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:45.796273    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5" event={"ID":"9c0bed99-7109-4dd2-87be-62d46d41a252","Type":"ContainerStarted","Data":"2b69d8fc899473f57cb5414e6effc811630494eeb99e70c59dc97867e208ee8f"}
Jul 17 11:01:51.321657 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.321566    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:01:51.391530 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.391490    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd64557b8-4zbhx"]
Jul 17 11:01:51.395900 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.395864    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:01:51.396053 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.395965    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd64557b8-4zbhx"
Jul 17 11:01:51.398609 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.398588    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-678fl\""
Jul 17 11:01:51.399916 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.399873    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 17 11:01:51.399916 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.399881    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 17 11:01:51.400095 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.399925    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 17 11:01:51.400095 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.399926    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 17 11:01:51.407277 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.407241    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd64557b8-4zbhx"]
Jul 17 11:01:51.431357 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.431332    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bddbcd75-lx542"]
Jul 17 11:01:51.434449 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.434427    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bddbcd75-lx542"
Jul 17 11:01:51.437488 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.437467    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 17 11:01:51.450996 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.450971    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bddbcd75-lx542"]
Jul 17 11:01:51.492338 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.492304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/852e2b5d-36d1-4338-90df-ed09edc75b81-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-dd64557b8-4zbhx\" (UID: \"852e2b5d-36d1-4338-90df-ed09edc75b81\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd64557b8-4zbhx"
Jul 17 11:01:51.492496 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.492357    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66kg5\" (UniqueName: \"kubernetes.io/projected/852e2b5d-36d1-4338-90df-ed09edc75b81-kube-api-access-66kg5\") pod \"managed-serviceaccount-addon-agent-dd64557b8-4zbhx\" (UID: \"852e2b5d-36d1-4338-90df-ed09edc75b81\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd64557b8-4zbhx"
Jul 17 11:01:51.516310 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.516285    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:01:51.593567 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.593539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66kg5\" (UniqueName: \"kubernetes.io/projected/852e2b5d-36d1-4338-90df-ed09edc75b81-kube-api-access-66kg5\") pod \"managed-serviceaccount-addon-agent-dd64557b8-4zbhx\" (UID: \"852e2b5d-36d1-4338-90df-ed09edc75b81\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd64557b8-4zbhx"
Jul 17 11:01:51.593686 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.593592    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ae80-abac-43a0-813a-f7682740fc13-tmp\") pod \"klusterlet-addon-workmgr-56bddbcd75-lx542\" (UID: \"ccc8ae80-abac-43a0-813a-f7682740fc13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bddbcd75-lx542"
Jul 17 11:01:51.593686 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.593628    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/852e2b5d-36d1-4338-90df-ed09edc75b81-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-dd64557b8-4zbhx\" (UID: \"852e2b5d-36d1-4338-90df-ed09edc75b81\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd64557b8-4zbhx"
Jul 17 11:01:51.593686 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.593650    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ccc8ae80-abac-43a0-813a-f7682740fc13-klusterlet-config\") pod \"klusterlet-addon-workmgr-56bddbcd75-lx542\" (UID: \"ccc8ae80-abac-43a0-813a-f7682740fc13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bddbcd75-lx542"
Jul 17 11:01:51.593686 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.593669    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2rs7\" (UniqueName: \"kubernetes.io/projected/ccc8ae80-abac-43a0-813a-f7682740fc13-kube-api-access-f2rs7\") pod \"klusterlet-addon-workmgr-56bddbcd75-lx542\" (UID: \"ccc8ae80-abac-43a0-813a-f7682740fc13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bddbcd75-lx542"
Jul 17 11:01:51.596195 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.596171    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/852e2b5d-36d1-4338-90df-ed09edc75b81-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-dd64557b8-4zbhx\" (UID: \"852e2b5d-36d1-4338-90df-ed09edc75b81\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd64557b8-4zbhx"
Jul 17 11:01:51.603113 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.603089    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66kg5\" (UniqueName: \"kubernetes.io/projected/852e2b5d-36d1-4338-90df-ed09edc75b81-kube-api-access-66kg5\") pod \"managed-serviceaccount-addon-agent-dd64557b8-4zbhx\" (UID: \"852e2b5d-36d1-4338-90df-ed09edc75b81\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd64557b8-4zbhx"
Jul 17 11:01:51.695086 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.695039    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2rs7\" (UniqueName: \"kubernetes.io/projected/ccc8ae80-abac-43a0-813a-f7682740fc13-kube-api-access-f2rs7\") pod \"klusterlet-addon-workmgr-56bddbcd75-lx542\" (UID: \"ccc8ae80-abac-43a0-813a-f7682740fc13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bddbcd75-lx542"
Jul 17 11:01:51.695220 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.695134    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ae80-abac-43a0-813a-f7682740fc13-tmp\") pod \"klusterlet-addon-workmgr-56bddbcd75-lx542\" (UID: \"ccc8ae80-abac-43a0-813a-f7682740fc13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bddbcd75-lx542"
Jul 17 11:01:51.695220 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.695190    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ccc8ae80-abac-43a0-813a-f7682740fc13-klusterlet-config\") pod \"klusterlet-addon-workmgr-56bddbcd75-lx542\" (UID: \"ccc8ae80-abac-43a0-813a-f7682740fc13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bddbcd75-lx542"
Jul 17 11:01:51.695513 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.695493    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccc8ae80-abac-43a0-813a-f7682740fc13-tmp\") pod \"klusterlet-addon-workmgr-56bddbcd75-lx542\" (UID: \"ccc8ae80-abac-43a0-813a-f7682740fc13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bddbcd75-lx542"
Jul 17 11:01:51.697510 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.697485    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ccc8ae80-abac-43a0-813a-f7682740fc13-klusterlet-config\") pod \"klusterlet-addon-workmgr-56bddbcd75-lx542\" (UID: \"ccc8ae80-abac-43a0-813a-f7682740fc13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bddbcd75-lx542"
Jul 17 11:01:51.704717 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.704693    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2rs7\" (UniqueName: \"kubernetes.io/projected/ccc8ae80-abac-43a0-813a-f7682740fc13-kube-api-access-f2rs7\") pod \"klusterlet-addon-workmgr-56bddbcd75-lx542\" (UID: \"ccc8ae80-abac-43a0-813a-f7682740fc13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bddbcd75-lx542"
Jul 17 11:01:51.715660 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.715642    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd64557b8-4zbhx"
Jul 17 11:01:51.744676 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.744646    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bddbcd75-lx542"
Jul 17 11:01:51.826592 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.826558    2563 generic.go:356] "Generic (PLEG): container finished" podID="9c0bed99-7109-4dd2-87be-62d46d41a252" containerID="d84542038c90fc13dc2057ca4a3d373e7da5a73984c4f34be9b7db85745bc201" exitCode=0
Jul 17 11:01:51.826753 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.826631    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5" event={"ID":"9c0bed99-7109-4dd2-87be-62d46d41a252","Type":"ContainerDied","Data":"d84542038c90fc13dc2057ca4a3d373e7da5a73984c4f34be9b7db85745bc201"}
Jul 17 11:01:51.857055 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.856957    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd64557b8-4zbhx"]
Jul 17 11:01:51.859571 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:01:51.859545    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod852e2b5d_36d1_4338_90df_ed09edc75b81.slice/crio-70ad88586e3c31f26bf9e56f83b74cc1ce1ff2294c0ae2176694ab85c3fe3e78 WatchSource:0}: Error finding container 70ad88586e3c31f26bf9e56f83b74cc1ce1ff2294c0ae2176694ab85c3fe3e78: Status 404 returned error can't find the container with id 70ad88586e3c31f26bf9e56f83b74cc1ce1ff2294c0ae2176694ab85c3fe3e78
Jul 17 11:01:51.886670 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:51.886594    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bddbcd75-lx542"]
Jul 17 11:01:51.889084 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:01:51.889041    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc8ae80_abac_43a0_813a_f7682740fc13.slice/crio-788fbc88bfba9ce3f0d5d0a036a87ae1214bedee53206bdd4bfc2ebf6fe4dddc WatchSource:0}: Error finding container 788fbc88bfba9ce3f0d5d0a036a87ae1214bedee53206bdd4bfc2ebf6fe4dddc: Status 404 returned error can't find the container with id 788fbc88bfba9ce3f0d5d0a036a87ae1214bedee53206bdd4bfc2ebf6fe4dddc
Jul 17 11:01:52.833787 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:52.833727    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd64557b8-4zbhx" event={"ID":"852e2b5d-36d1-4338-90df-ed09edc75b81","Type":"ContainerStarted","Data":"70ad88586e3c31f26bf9e56f83b74cc1ce1ff2294c0ae2176694ab85c3fe3e78"}
Jul 17 11:01:52.835318 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:52.835287    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bddbcd75-lx542" event={"ID":"ccc8ae80-abac-43a0-813a-f7682740fc13","Type":"ContainerStarted","Data":"788fbc88bfba9ce3f0d5d0a036a87ae1214bedee53206bdd4bfc2ebf6fe4dddc"}
Jul 17 11:01:54.357150 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:54.357111    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:01:54.459844 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:54.459786    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:01:55.850742 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:55.850711    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bddbcd75-lx542" event={"ID":"ccc8ae80-abac-43a0-813a-f7682740fc13","Type":"ContainerStarted","Data":"843a6ce26084cdf91f440e0ac44293bf5a3dc26783b23af94459f923d7e91b73"}
Jul 17 11:01:55.851198 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:55.850871    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bddbcd75-lx542"
Jul 17 11:01:55.852370 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:55.852349    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bddbcd75-lx542"
Jul 17 11:01:55.852825 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:55.852803    2563 generic.go:356] "Generic (PLEG): container finished" podID="9c0bed99-7109-4dd2-87be-62d46d41a252" containerID="42c469d17c5d1317ea792471977545bbd10423cc44de4561cbb12f5dedc38897" exitCode=0
Jul 17 11:01:55.852913 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:55.852848    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5" event={"ID":"9c0bed99-7109-4dd2-87be-62d46d41a252","Type":"ContainerDied","Data":"42c469d17c5d1317ea792471977545bbd10423cc44de4561cbb12f5dedc38897"}
Jul 17 11:01:55.870961 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:01:55.870914    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bddbcd75-lx542" podStartSLOduration=1.019212165 podStartE2EDuration="4.870898807s" totalImagesPullingTime="3.851686642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:01:55.869441136 +0000 UTC m=+270.798980773" watchObservedRunningTime="2026-07-17 11:01:55.870898807 +0000 UTC m=+270.800438478"
Jul 17 11:02:02.883105 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:02.883052    2563 generic.go:356] "Generic (PLEG): container finished" podID="9c0bed99-7109-4dd2-87be-62d46d41a252" containerID="ed3242a31c161723328e13e7d2037c7a7e95407baec1d2993cb5c11fcc795661" exitCode=0
Jul 17 11:02:02.883487 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:02.883135    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5" event={"ID":"9c0bed99-7109-4dd2-87be-62d46d41a252","Type":"ContainerDied","Data":"ed3242a31c161723328e13e7d2037c7a7e95407baec1d2993cb5c11fcc795661"}
Jul 17 11:02:04.031470 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:04.031436    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5"
Jul 17 11:02:04.092287 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:04.092245    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mpdm\" (UniqueName: \"kubernetes.io/projected/9c0bed99-7109-4dd2-87be-62d46d41a252-kube-api-access-7mpdm\") pod \"9c0bed99-7109-4dd2-87be-62d46d41a252\" (UID: \"9c0bed99-7109-4dd2-87be-62d46d41a252\") "
Jul 17 11:02:04.092287 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:04.092298    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c0bed99-7109-4dd2-87be-62d46d41a252-util\") pod \"9c0bed99-7109-4dd2-87be-62d46d41a252\" (UID: \"9c0bed99-7109-4dd2-87be-62d46d41a252\") "
Jul 17 11:02:04.092509 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:04.092325    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c0bed99-7109-4dd2-87be-62d46d41a252-bundle\") pod \"9c0bed99-7109-4dd2-87be-62d46d41a252\" (UID: \"9c0bed99-7109-4dd2-87be-62d46d41a252\") "
Jul 17 11:02:04.093005 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:04.092978    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0bed99-7109-4dd2-87be-62d46d41a252-bundle" (OuterVolumeSpecName: "bundle") pod "9c0bed99-7109-4dd2-87be-62d46d41a252" (UID: "9c0bed99-7109-4dd2-87be-62d46d41a252"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:02:04.094458 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:04.094433    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c0bed99-7109-4dd2-87be-62d46d41a252-kube-api-access-7mpdm" (OuterVolumeSpecName: "kube-api-access-7mpdm") pod "9c0bed99-7109-4dd2-87be-62d46d41a252" (UID: "9c0bed99-7109-4dd2-87be-62d46d41a252"). InnerVolumeSpecName "kube-api-access-7mpdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 11:02:04.098996 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:04.098969    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0bed99-7109-4dd2-87be-62d46d41a252-util" (OuterVolumeSpecName: "util") pod "9c0bed99-7109-4dd2-87be-62d46d41a252" (UID: "9c0bed99-7109-4dd2-87be-62d46d41a252"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:02:04.192983 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:04.192902    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mpdm\" (UniqueName: \"kubernetes.io/projected/9c0bed99-7109-4dd2-87be-62d46d41a252-kube-api-access-7mpdm\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:02:04.192983 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:04.192936    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c0bed99-7109-4dd2-87be-62d46d41a252-util\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:02:04.192983 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:04.192945    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c0bed99-7109-4dd2-87be-62d46d41a252-bundle\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:02:04.892736 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:04.892707    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5"
Jul 17 11:02:04.892930 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:04.892705    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q42r5" event={"ID":"9c0bed99-7109-4dd2-87be-62d46d41a252","Type":"ContainerDied","Data":"2b69d8fc899473f57cb5414e6effc811630494eeb99e70c59dc97867e208ee8f"}
Jul 17 11:02:04.892930 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:04.892838    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b69d8fc899473f57cb5414e6effc811630494eeb99e70c59dc97867e208ee8f"
Jul 17 11:02:04.894213 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:04.894180    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd64557b8-4zbhx" event={"ID":"852e2b5d-36d1-4338-90df-ed09edc75b81","Type":"ContainerStarted","Data":"2ececdc4bd012d46a5d8851c51a406ddb66b6febfe63dfcfc5b863d6937c2f2e"}
Jul 17 11:02:04.913619 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:04.913574    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd64557b8-4zbhx" podStartSLOduration=1.8869176319999998 podStartE2EDuration="13.913559245s" totalImagesPullingTime="12.026641613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:02:04.911810266 +0000 UTC m=+279.841349906" watchObservedRunningTime="2026-07-17 11:02:04.913559245 +0000 UTC m=+279.843098884"
Jul 17 11:02:10.811294 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:10.811257    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:11.717715 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:11.717673    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:14.411395 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:14.411361    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:14.510993 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:14.510947    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:16.301381 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:16.301302    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:16.432189 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:16.432139    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:16.597262 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:16.597183    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq"]
Jul 17 11:02:16.597481 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:16.597468    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c0bed99-7109-4dd2-87be-62d46d41a252" containerName="extract"
Jul 17 11:02:16.597581 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:16.597482    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c0bed99-7109-4dd2-87be-62d46d41a252" containerName="extract"
Jul 17 11:02:16.597581 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:16.597497    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c0bed99-7109-4dd2-87be-62d46d41a252" containerName="pull"
Jul 17 11:02:16.597581 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:16.597503    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c0bed99-7109-4dd2-87be-62d46d41a252" containerName="pull"
Jul 17 11:02:16.597581 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:16.597510    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c0bed99-7109-4dd2-87be-62d46d41a252" containerName="util"
Jul 17 11:02:16.597581 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:16.597517    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c0bed99-7109-4dd2-87be-62d46d41a252" containerName="util"
Jul 17 11:02:16.597581 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:16.597565    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c0bed99-7109-4dd2-87be-62d46d41a252" containerName="extract"
Jul 17 11:02:16.599945 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:16.599927    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq"
Jul 17 11:02:16.608143 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:16.608121    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gxg5n\""
Jul 17 11:02:16.609277 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:16.609261    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 17 11:02:16.609921 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:16.609903    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 17 11:02:16.618286 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:16.618260    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq"]
Jul 17 11:02:16.693009 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:16.692968    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93bc8afb-2d80-43c6-b71c-9aa27ba886e8-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq\" (UID: \"93bc8afb-2d80-43c6-b71c-9aa27ba886e8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq"
Jul 17 11:02:16.693197 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:16.693022    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb9w4\" (UniqueName: \"kubernetes.io/projected/93bc8afb-2d80-43c6-b71c-9aa27ba886e8-kube-api-access-jb9w4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq\" (UID: \"93bc8afb-2d80-43c6-b71c-9aa27ba886e8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq"
Jul 17 11:02:16.693197 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:16.693096    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93bc8afb-2d80-43c6-b71c-9aa27ba886e8-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq\" (UID: \"93bc8afb-2d80-43c6-b71c-9aa27ba886e8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq"
Jul 17 11:02:16.697038 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:16.697011    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:16.793587 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:16.793538    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb9w4\" (UniqueName: \"kubernetes.io/projected/93bc8afb-2d80-43c6-b71c-9aa27ba886e8-kube-api-access-jb9w4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq\" (UID: \"93bc8afb-2d80-43c6-b71c-9aa27ba886e8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq"
Jul 17 11:02:16.793587 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:16.793597    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93bc8afb-2d80-43c6-b71c-9aa27ba886e8-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq\" (UID: \"93bc8afb-2d80-43c6-b71c-9aa27ba886e8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq"
Jul 17 11:02:16.793817 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:16.793652    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93bc8afb-2d80-43c6-b71c-9aa27ba886e8-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq\" (UID: \"93bc8afb-2d80-43c6-b71c-9aa27ba886e8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq"
Jul 17 11:02:16.794011 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:16.793992    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93bc8afb-2d80-43c6-b71c-9aa27ba886e8-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq\" (UID: \"93bc8afb-2d80-43c6-b71c-9aa27ba886e8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq"
Jul 17 11:02:16.794092 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:16.794018    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93bc8afb-2d80-43c6-b71c-9aa27ba886e8-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq\" (UID: \"93bc8afb-2d80-43c6-b71c-9aa27ba886e8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq"
Jul 17 11:02:16.812456 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:16.812420    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb9w4\" (UniqueName: \"kubernetes.io/projected/93bc8afb-2d80-43c6-b71c-9aa27ba886e8-kube-api-access-jb9w4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq\" (UID: \"93bc8afb-2d80-43c6-b71c-9aa27ba886e8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq"
Jul 17 11:02:16.819905 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:16.819868    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:16.908323 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:16.908282    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq"
Jul 17 11:02:17.066688 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:02:17.066660    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93bc8afb_2d80_43c6_b71c_9aa27ba886e8.slice/crio-740a6c95ed4b68a91486e003a90360fe5eaab4c8d2fe541c2f1edebf74652f16 WatchSource:0}: Error finding container 740a6c95ed4b68a91486e003a90360fe5eaab4c8d2fe541c2f1edebf74652f16: Status 404 returned error can't find the container with id 740a6c95ed4b68a91486e003a90360fe5eaab4c8d2fe541c2f1edebf74652f16
Jul 17 11:02:17.083263 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:17.083236    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq"]
Jul 17 11:02:17.947974 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:17.947937    2563 generic.go:356] "Generic (PLEG): container finished" podID="93bc8afb-2d80-43c6-b71c-9aa27ba886e8" containerID="b903eaf6e03a4797b1e9afa5e702df9eecf72d94743df6e12aa3b9fd8919ad25" exitCode=0
Jul 17 11:02:17.948390 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:17.948024    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq" event={"ID":"93bc8afb-2d80-43c6-b71c-9aa27ba886e8","Type":"ContainerDied","Data":"b903eaf6e03a4797b1e9afa5e702df9eecf72d94743df6e12aa3b9fd8919ad25"}
Jul 17 11:02:17.948390 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:17.948094    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq" event={"ID":"93bc8afb-2d80-43c6-b71c-9aa27ba886e8","Type":"ContainerStarted","Data":"740a6c95ed4b68a91486e003a90360fe5eaab4c8d2fe541c2f1edebf74652f16"}
Jul 17 11:02:18.930036 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:18.929993    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:19.361807 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:19.361726    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:19.959403 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:19.959369    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq" event={"ID":"93bc8afb-2d80-43c6-b71c-9aa27ba886e8","Type":"ContainerStarted","Data":"660f58e073eb25c6f4595792efb35927e0dd5eef1a85a76b1c0c5a7a72e0c35d"}
Jul 17 11:02:20.964402 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:20.964363    2563 generic.go:356] "Generic (PLEG): container finished" podID="93bc8afb-2d80-43c6-b71c-9aa27ba886e8" containerID="660f58e073eb25c6f4595792efb35927e0dd5eef1a85a76b1c0c5a7a72e0c35d" exitCode=0
Jul 17 11:02:20.964774 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:20.964436    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq" event={"ID":"93bc8afb-2d80-43c6-b71c-9aa27ba886e8","Type":"ContainerDied","Data":"660f58e073eb25c6f4595792efb35927e0dd5eef1a85a76b1c0c5a7a72e0c35d"}
Jul 17 11:02:21.877411 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:21.877380    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:21.970906 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:21.970873    2563 generic.go:356] "Generic (PLEG): container finished" podID="93bc8afb-2d80-43c6-b71c-9aa27ba886e8" containerID="155e1b065e25efdf23b66e70816a88bd6e6e3c9a7cd4949e4c2413557fd97ca6" exitCode=0
Jul 17 11:02:21.971304 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:21.970928    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq" event={"ID":"93bc8afb-2d80-43c6-b71c-9aa27ba886e8","Type":"ContainerDied","Data":"155e1b065e25efdf23b66e70816a88bd6e6e3c9a7cd4949e4c2413557fd97ca6"}
Jul 17 11:02:22.019388 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:22.019357    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:23.097254 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.097229    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq"
Jul 17 11:02:23.147115 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.147047    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93bc8afb-2d80-43c6-b71c-9aa27ba886e8-util\") pod \"93bc8afb-2d80-43c6-b71c-9aa27ba886e8\" (UID: \"93bc8afb-2d80-43c6-b71c-9aa27ba886e8\") "
Jul 17 11:02:23.147302 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.147147    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jb9w4\" (UniqueName: \"kubernetes.io/projected/93bc8afb-2d80-43c6-b71c-9aa27ba886e8-kube-api-access-jb9w4\") pod \"93bc8afb-2d80-43c6-b71c-9aa27ba886e8\" (UID: \"93bc8afb-2d80-43c6-b71c-9aa27ba886e8\") "
Jul 17 11:02:23.147302 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.147196    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93bc8afb-2d80-43c6-b71c-9aa27ba886e8-bundle\") pod \"93bc8afb-2d80-43c6-b71c-9aa27ba886e8\" (UID: \"93bc8afb-2d80-43c6-b71c-9aa27ba886e8\") "
Jul 17 11:02:23.147605 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.147574    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93bc8afb-2d80-43c6-b71c-9aa27ba886e8-bundle" (OuterVolumeSpecName: "bundle") pod "93bc8afb-2d80-43c6-b71c-9aa27ba886e8" (UID: "93bc8afb-2d80-43c6-b71c-9aa27ba886e8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:02:23.149278 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.149257    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93bc8afb-2d80-43c6-b71c-9aa27ba886e8-kube-api-access-jb9w4" (OuterVolumeSpecName: "kube-api-access-jb9w4") pod "93bc8afb-2d80-43c6-b71c-9aa27ba886e8" (UID: "93bc8afb-2d80-43c6-b71c-9aa27ba886e8"). InnerVolumeSpecName "kube-api-access-jb9w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 11:02:23.154377 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.154353    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93bc8afb-2d80-43c6-b71c-9aa27ba886e8-util" (OuterVolumeSpecName: "util") pod "93bc8afb-2d80-43c6-b71c-9aa27ba886e8" (UID: "93bc8afb-2d80-43c6-b71c-9aa27ba886e8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:02:23.248436 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.248330    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93bc8afb-2d80-43c6-b71c-9aa27ba886e8-util\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:02:23.248436 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.248381    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb9w4\" (UniqueName: \"kubernetes.io/projected/93bc8afb-2d80-43c6-b71c-9aa27ba886e8-kube-api-access-jb9w4\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:02:23.248436 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.248391    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93bc8afb-2d80-43c6-b71c-9aa27ba886e8-bundle\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:02:23.421322 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.421283    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-spkzb"]
Jul 17 11:02:23.421586 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.421574    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93bc8afb-2d80-43c6-b71c-9aa27ba886e8" containerName="util"
Jul 17 11:02:23.421650 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.421587    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="93bc8afb-2d80-43c6-b71c-9aa27ba886e8" containerName="util"
Jul 17 11:02:23.421650 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.421601    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93bc8afb-2d80-43c6-b71c-9aa27ba886e8" containerName="pull"
Jul 17 11:02:23.421650 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.421606    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="93bc8afb-2d80-43c6-b71c-9aa27ba886e8" containerName="pull"
Jul 17 11:02:23.421650 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.421617    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93bc8afb-2d80-43c6-b71c-9aa27ba886e8" containerName="extract"
Jul 17 11:02:23.421650 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.421623    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="93bc8afb-2d80-43c6-b71c-9aa27ba886e8" containerName="extract"
Jul 17 11:02:23.421810 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.421670    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="93bc8afb-2d80-43c6-b71c-9aa27ba886e8" containerName="extract"
Jul 17 11:02:23.423805 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.423787    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-spkzb"
Jul 17 11:02:23.426690 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.426671    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 17 11:02:23.426791 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.426757    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 17 11:02:23.427178 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.427164    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-nfkth\""
Jul 17 11:02:23.436652 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.436623    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-spkzb"]
Jul 17 11:02:23.551428 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.551334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f2a7ef4-de4d-461b-bba6-77b48ff4cc69-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-spkzb\" (UID: \"0f2a7ef4-de4d-461b-bba6-77b48ff4cc69\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-spkzb"
Jul 17 11:02:23.551428 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.551377    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28lf4\" (UniqueName: \"kubernetes.io/projected/0f2a7ef4-de4d-461b-bba6-77b48ff4cc69-kube-api-access-28lf4\") pod \"cert-manager-cainjector-559cdd8c87-spkzb\" (UID: \"0f2a7ef4-de4d-461b-bba6-77b48ff4cc69\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-spkzb"
Jul 17 11:02:23.652683 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.652647    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f2a7ef4-de4d-461b-bba6-77b48ff4cc69-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-spkzb\" (UID: \"0f2a7ef4-de4d-461b-bba6-77b48ff4cc69\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-spkzb"
Jul 17 11:02:23.652683 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.652688    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28lf4\" (UniqueName: \"kubernetes.io/projected/0f2a7ef4-de4d-461b-bba6-77b48ff4cc69-kube-api-access-28lf4\") pod \"cert-manager-cainjector-559cdd8c87-spkzb\" (UID: \"0f2a7ef4-de4d-461b-bba6-77b48ff4cc69\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-spkzb"
Jul 17 11:02:23.662553 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.662520    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f2a7ef4-de4d-461b-bba6-77b48ff4cc69-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-spkzb\" (UID: \"0f2a7ef4-de4d-461b-bba6-77b48ff4cc69\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-spkzb"
Jul 17 11:02:23.662692 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.662620    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28lf4\" (UniqueName: \"kubernetes.io/projected/0f2a7ef4-de4d-461b-bba6-77b48ff4cc69-kube-api-access-28lf4\") pod \"cert-manager-cainjector-559cdd8c87-spkzb\" (UID: \"0f2a7ef4-de4d-461b-bba6-77b48ff4cc69\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-spkzb"
Jul 17 11:02:23.732681 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.732643    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-spkzb"
Jul 17 11:02:23.862204 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.862173    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-spkzb"]
Jul 17 11:02:23.865158 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:02:23.865124    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f2a7ef4_de4d_461b_bba6_77b48ff4cc69.slice/crio-1d6e2040dded7c8d5de94792ad74cf22c637feceb32d05313394f6a877826241 WatchSource:0}: Error finding container 1d6e2040dded7c8d5de94792ad74cf22c637feceb32d05313394f6a877826241: Status 404 returned error can't find the container with id 1d6e2040dded7c8d5de94792ad74cf22c637feceb32d05313394f6a877826241
Jul 17 11:02:23.979095 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.979036    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-spkzb" event={"ID":"0f2a7ef4-de4d-461b-bba6-77b48ff4cc69","Type":"ContainerStarted","Data":"1d6e2040dded7c8d5de94792ad74cf22c637feceb32d05313394f6a877826241"}
Jul 17 11:02:23.980796 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.980768    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq" event={"ID":"93bc8afb-2d80-43c6-b71c-9aa27ba886e8","Type":"ContainerDied","Data":"740a6c95ed4b68a91486e003a90360fe5eaab4c8d2fe541c2f1edebf74652f16"}
Jul 17 11:02:23.980922 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.980805    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs4hfq"
Jul 17 11:02:23.980922 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:23.980811    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="740a6c95ed4b68a91486e003a90360fe5eaab4c8d2fe541c2f1edebf74652f16"
Jul 17 11:02:25.457086 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:25.456978    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nj5mj_55d1de37-8635-4de8-ab59-4ebfd0346a40/ovn-acl-logging/0.log"
Jul 17 11:02:25.463745 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:25.463720    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nj5mj_55d1de37-8635-4de8-ab59-4ebfd0346a40/ovn-acl-logging/0.log"
Jul 17 11:02:25.465603 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:25.465573    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 17 11:02:27.000657 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:27.000617    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-spkzb" event={"ID":"0f2a7ef4-de4d-461b-bba6-77b48ff4cc69","Type":"ContainerStarted","Data":"e7dc3c1cd34d7f6ae3029a7a915c84d87825e3d971036510236bb1d25592bc10"}
Jul 17 11:02:27.010132 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:27.010042    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:27.030460 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:27.030412    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-spkzb" podStartSLOduration=1.363923199 podStartE2EDuration="4.030396163s" totalImagesPullingTime="2.666472964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:02:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:02:27.028396607 +0000 UTC m=+301.957936247" watchObservedRunningTime="2026-07-17 11:02:27.030396163 +0000 UTC m=+301.959935803"
Jul 17 11:02:27.330119 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:27.329984    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:27.434302 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:27.434267    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:27.531311 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:27.531272    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:29.672893 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:29.672839    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zz7jl"]
Jul 17 11:02:29.675120 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:29.675104    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zz7jl"
Jul 17 11:02:29.677749 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:29.677724    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 17 11:02:29.678650 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:29.678632    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-992hw\""
Jul 17 11:02:29.678756 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:29.678634    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 17 11:02:29.686915 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:29.686894    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zz7jl"]
Jul 17 11:02:29.806637 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:29.806603    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3918ecd3-a833-43da-988f-df80290a9407-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zz7jl\" (UID: \"3918ecd3-a833-43da-988f-df80290a9407\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zz7jl"
Jul 17 11:02:29.806637 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:29.806643    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr6w4\" (UniqueName: \"kubernetes.io/projected/3918ecd3-a833-43da-988f-df80290a9407-kube-api-access-pr6w4\") pod \"openshift-lws-operator-fd8ccff4c-zz7jl\" (UID: \"3918ecd3-a833-43da-988f-df80290a9407\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zz7jl"
Jul 17 11:02:29.907795 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:29.907744    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3918ecd3-a833-43da-988f-df80290a9407-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zz7jl\" (UID: \"3918ecd3-a833-43da-988f-df80290a9407\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zz7jl"
Jul 17 11:02:29.907795 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:29.907806    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr6w4\" (UniqueName: \"kubernetes.io/projected/3918ecd3-a833-43da-988f-df80290a9407-kube-api-access-pr6w4\") pod \"openshift-lws-operator-fd8ccff4c-zz7jl\" (UID: \"3918ecd3-a833-43da-988f-df80290a9407\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zz7jl"
Jul 17 11:02:29.908186 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:29.908165    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3918ecd3-a833-43da-988f-df80290a9407-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zz7jl\" (UID: \"3918ecd3-a833-43da-988f-df80290a9407\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zz7jl"
Jul 17 11:02:29.918294 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:29.918263    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr6w4\" (UniqueName: \"kubernetes.io/projected/3918ecd3-a833-43da-988f-df80290a9407-kube-api-access-pr6w4\") pod \"openshift-lws-operator-fd8ccff4c-zz7jl\" (UID: \"3918ecd3-a833-43da-988f-df80290a9407\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zz7jl"
Jul 17 11:02:29.984113 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:29.984006    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zz7jl"
Jul 17 11:02:30.122842 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:30.122808    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zz7jl"]
Jul 17 11:02:30.126243 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:02:30.126211    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3918ecd3_a833_43da_988f_df80290a9407.slice/crio-e205f56b1b28dc4e51ee68b6ce1eb47630961278dd8aac0a4769603a0188f5a7 WatchSource:0}: Error finding container e205f56b1b28dc4e51ee68b6ce1eb47630961278dd8aac0a4769603a0188f5a7: Status 404 returned error can't find the container with id e205f56b1b28dc4e51ee68b6ce1eb47630961278dd8aac0a4769603a0188f5a7
Jul 17 11:02:30.128279 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:30.128262    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 11:02:31.020263 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:31.020216    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zz7jl" event={"ID":"3918ecd3-a833-43da-988f-df80290a9407","Type":"ContainerStarted","Data":"e205f56b1b28dc4e51ee68b6ce1eb47630961278dd8aac0a4769603a0188f5a7"}
Jul 17 11:02:33.029731 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:33.029696    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zz7jl" event={"ID":"3918ecd3-a833-43da-988f-df80290a9407","Type":"ContainerStarted","Data":"8b0a179ac1ad189fe5366422db10ee34fe8deba1a3f8f7b5f10f056cc75166fd"}
Jul 17 11:02:33.050485 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:33.050434    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zz7jl" podStartSLOduration=2.191061798 podStartE2EDuration="4.05041733s" totalImagesPullingTime="1.859355532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:02:33.049679553 +0000 UTC m=+307.979219194" watchObservedRunningTime="2026-07-17 11:02:33.05041733 +0000 UTC m=+307.979956969"
Jul 17 11:02:33.769455 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:33.769424    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:33.871043 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:33.871007    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:35.713462 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:35.713426    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:37.361406 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:37.361342    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq"]
Jul 17 11:02:37.382878 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:37.382846    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq"]
Jul 17 11:02:37.383046 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:37.382960    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq"
Jul 17 11:02:37.385574 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:37.385552    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 17 11:02:37.386511 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:37.386495    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 17 11:02:37.386595 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:37.386499    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gxg5n\""
Jul 17 11:02:37.470959 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:37.470920    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ce53ce5-68e7-4358-b410-ee7904cacc6c-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq\" (UID: \"0ce53ce5-68e7-4358-b410-ee7904cacc6c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq"
Jul 17 11:02:37.471171 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:37.470983    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ce53ce5-68e7-4358-b410-ee7904cacc6c-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq\" (UID: \"0ce53ce5-68e7-4358-b410-ee7904cacc6c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq"
Jul 17 11:02:37.471171 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:37.471005    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qkrk\" (UniqueName: \"kubernetes.io/projected/0ce53ce5-68e7-4358-b410-ee7904cacc6c-kube-api-access-7qkrk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq\" (UID: \"0ce53ce5-68e7-4358-b410-ee7904cacc6c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq"
Jul 17 11:02:37.571968 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:37.571922    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ce53ce5-68e7-4358-b410-ee7904cacc6c-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq\" (UID: \"0ce53ce5-68e7-4358-b410-ee7904cacc6c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq"
Jul 17 11:02:37.572204 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:37.572019    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ce53ce5-68e7-4358-b410-ee7904cacc6c-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq\" (UID: \"0ce53ce5-68e7-4358-b410-ee7904cacc6c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq"
Jul 17 11:02:37.572204 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:37.572045    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qkrk\" (UniqueName: \"kubernetes.io/projected/0ce53ce5-68e7-4358-b410-ee7904cacc6c-kube-api-access-7qkrk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq\" (UID: \"0ce53ce5-68e7-4358-b410-ee7904cacc6c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq"
Jul 17 11:02:37.572454 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:37.572427    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ce53ce5-68e7-4358-b410-ee7904cacc6c-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq\" (UID: \"0ce53ce5-68e7-4358-b410-ee7904cacc6c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq"
Jul 17 11:02:37.572521 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:37.572472    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ce53ce5-68e7-4358-b410-ee7904cacc6c-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq\" (UID: \"0ce53ce5-68e7-4358-b410-ee7904cacc6c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq"
Jul 17 11:02:37.582586 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:37.582559    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qkrk\" (UniqueName: \"kubernetes.io/projected/0ce53ce5-68e7-4358-b410-ee7904cacc6c-kube-api-access-7qkrk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq\" (UID: \"0ce53ce5-68e7-4358-b410-ee7904cacc6c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq"
Jul 17 11:02:37.692160 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:37.692125    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq"
Jul 17 11:02:37.857119 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:02:37.857090    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ce53ce5_68e7_4358_b410_ee7904cacc6c.slice/crio-4f9bc75a2e672250a1f4b1c375797e422338f51d7cc3e4d8a246761209d91e1a WatchSource:0}: Error finding container 4f9bc75a2e672250a1f4b1c375797e422338f51d7cc3e4d8a246761209d91e1a: Status 404 returned error can't find the container with id 4f9bc75a2e672250a1f4b1c375797e422338f51d7cc3e4d8a246761209d91e1a
Jul 17 11:02:37.858410 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:37.858387    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq"]
Jul 17 11:02:38.052475 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:38.052443    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ce53ce5-68e7-4358-b410-ee7904cacc6c" containerID="3f902e29d88e83222f5292d7dc6edff83bb5063e2781d70bb5420ec736ec06a9" exitCode=0
Jul 17 11:02:38.052634 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:38.052530    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq" event={"ID":"0ce53ce5-68e7-4358-b410-ee7904cacc6c","Type":"ContainerDied","Data":"3f902e29d88e83222f5292d7dc6edff83bb5063e2781d70bb5420ec736ec06a9"}
Jul 17 11:02:38.052634 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:38.052579    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq" event={"ID":"0ce53ce5-68e7-4358-b410-ee7904cacc6c","Type":"ContainerStarted","Data":"4f9bc75a2e672250a1f4b1c375797e422338f51d7cc3e4d8a246761209d91e1a"}
Jul 17 11:02:40.063390 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:40.063359    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ce53ce5-68e7-4358-b410-ee7904cacc6c" containerID="e3121fe0314d55374bd36f69abfe98bb64726de1def4d7a4f0dbd721a899ee9d" exitCode=0
Jul 17 11:02:40.063843 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:40.063445    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq" event={"ID":"0ce53ce5-68e7-4358-b410-ee7904cacc6c","Type":"ContainerDied","Data":"e3121fe0314d55374bd36f69abfe98bb64726de1def4d7a4f0dbd721a899ee9d"}
Jul 17 11:02:41.069661 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:41.069629    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ce53ce5-68e7-4358-b410-ee7904cacc6c" containerID="8c3e12f9f21abd2d81428efd539d5ec02b5f0cb8f4206874796e21db1a620dba" exitCode=0
Jul 17 11:02:41.070098 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:41.069711    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq" event={"ID":"0ce53ce5-68e7-4358-b410-ee7904cacc6c","Type":"ContainerDied","Data":"8c3e12f9f21abd2d81428efd539d5ec02b5f0cb8f4206874796e21db1a620dba"}
Jul 17 11:02:41.486623 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:41.486588    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:42.199793 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:42.199768    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq"
Jul 17 11:02:42.309811 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:42.309777    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qkrk\" (UniqueName: \"kubernetes.io/projected/0ce53ce5-68e7-4358-b410-ee7904cacc6c-kube-api-access-7qkrk\") pod \"0ce53ce5-68e7-4358-b410-ee7904cacc6c\" (UID: \"0ce53ce5-68e7-4358-b410-ee7904cacc6c\") "
Jul 17 11:02:42.309997 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:42.309837    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ce53ce5-68e7-4358-b410-ee7904cacc6c-util\") pod \"0ce53ce5-68e7-4358-b410-ee7904cacc6c\" (UID: \"0ce53ce5-68e7-4358-b410-ee7904cacc6c\") "
Jul 17 11:02:42.309997 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:42.309866    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ce53ce5-68e7-4358-b410-ee7904cacc6c-bundle\") pod \"0ce53ce5-68e7-4358-b410-ee7904cacc6c\" (UID: \"0ce53ce5-68e7-4358-b410-ee7904cacc6c\") "
Jul 17 11:02:42.310799 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:42.310766    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce53ce5-68e7-4358-b410-ee7904cacc6c-bundle" (OuterVolumeSpecName: "bundle") pod "0ce53ce5-68e7-4358-b410-ee7904cacc6c" (UID: "0ce53ce5-68e7-4358-b410-ee7904cacc6c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:02:42.311864 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:42.311828    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ce53ce5-68e7-4358-b410-ee7904cacc6c-kube-api-access-7qkrk" (OuterVolumeSpecName: "kube-api-access-7qkrk") pod "0ce53ce5-68e7-4358-b410-ee7904cacc6c" (UID: "0ce53ce5-68e7-4358-b410-ee7904cacc6c"). InnerVolumeSpecName "kube-api-access-7qkrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 11:02:42.315974 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:42.315952    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce53ce5-68e7-4358-b410-ee7904cacc6c-util" (OuterVolumeSpecName: "util") pod "0ce53ce5-68e7-4358-b410-ee7904cacc6c" (UID: "0ce53ce5-68e7-4358-b410-ee7904cacc6c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:02:42.411043 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:42.411011    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qkrk\" (UniqueName: \"kubernetes.io/projected/0ce53ce5-68e7-4358-b410-ee7904cacc6c-kube-api-access-7qkrk\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:02:42.411043 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:42.411043    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ce53ce5-68e7-4358-b410-ee7904cacc6c-util\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:02:42.411254 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:42.411053    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ce53ce5-68e7-4358-b410-ee7904cacc6c-bundle\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:02:43.078841 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:43.078797    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq" event={"ID":"0ce53ce5-68e7-4358-b410-ee7904cacc6c","Type":"ContainerDied","Data":"4f9bc75a2e672250a1f4b1c375797e422338f51d7cc3e4d8a246761209d91e1a"}
Jul 17 11:02:43.078841 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:43.078832    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsmspq"
Jul 17 11:02:43.078841 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:43.078847    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f9bc75a2e672250a1f4b1c375797e422338f51d7cc3e4d8a246761209d91e1a"
Jul 17 11:02:47.871647 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:47.871606    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:47.973247 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:47.973212    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:48.540813 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:48.540774    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:48.883840 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:48.883807    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:49.015329 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.015271    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:49.017400 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.017374    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cd97f7fc7-54tcg"]
Jul 17 11:02:49.017656 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.017645    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ce53ce5-68e7-4358-b410-ee7904cacc6c" containerName="util"
Jul 17 11:02:49.017696 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.017658    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ce53ce5-68e7-4358-b410-ee7904cacc6c" containerName="util"
Jul 17 11:02:49.017696 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.017667    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ce53ce5-68e7-4358-b410-ee7904cacc6c" containerName="extract"
Jul 17 11:02:49.017696 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.017672    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ce53ce5-68e7-4358-b410-ee7904cacc6c" containerName="extract"
Jul 17 11:02:49.017696 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.017684    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ce53ce5-68e7-4358-b410-ee7904cacc6c" containerName="pull"
Jul 17 11:02:49.017696 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.017689    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ce53ce5-68e7-4358-b410-ee7904cacc6c" containerName="pull"
Jul 17 11:02:49.017837 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.017738    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ce53ce5-68e7-4358-b410-ee7904cacc6c" containerName="extract"
Jul 17 11:02:49.019776 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.019761    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cd97f7fc7-54tcg"
Jul 17 11:02:49.023441 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.023424    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 17 11:02:49.024441 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.024421    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 17 11:02:49.024666 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.024650    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hx8rl\""
Jul 17 11:02:49.024961 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.024939    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 17 11:02:49.026268 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.025948    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 17 11:02:49.057913 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.057880    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cd97f7fc7-54tcg"]
Jul 17 11:02:49.172973 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.172884    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22f3c1ce-3171-49cf-8a07-4c0fcc2ae585-tmp\") pod \"opendatahub-operator-controller-manager-6cd97f7fc7-54tcg\" (UID: \"22f3c1ce-3171-49cf-8a07-4c0fcc2ae585\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd97f7fc7-54tcg"
Jul 17 11:02:49.172973 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.172921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/22f3c1ce-3171-49cf-8a07-4c0fcc2ae585-manifests\") pod \"opendatahub-operator-controller-manager-6cd97f7fc7-54tcg\" (UID: \"22f3c1ce-3171-49cf-8a07-4c0fcc2ae585\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd97f7fc7-54tcg"
Jul 17 11:02:49.172973 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.172948    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/22f3c1ce-3171-49cf-8a07-4c0fcc2ae585-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cd97f7fc7-54tcg\" (UID: \"22f3c1ce-3171-49cf-8a07-4c0fcc2ae585\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd97f7fc7-54tcg"
Jul 17 11:02:49.172973 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.172967    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zx7z\" (UniqueName: \"kubernetes.io/projected/22f3c1ce-3171-49cf-8a07-4c0fcc2ae585-kube-api-access-5zx7z\") pod \"opendatahub-operator-controller-manager-6cd97f7fc7-54tcg\" (UID: \"22f3c1ce-3171-49cf-8a07-4c0fcc2ae585\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd97f7fc7-54tcg"
Jul 17 11:02:49.173249 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.173052    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22f3c1ce-3171-49cf-8a07-4c0fcc2ae585-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cd97f7fc7-54tcg\" (UID: \"22f3c1ce-3171-49cf-8a07-4c0fcc2ae585\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd97f7fc7-54tcg"
Jul 17 11:02:49.273853 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.273810    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/22f3c1ce-3171-49cf-8a07-4c0fcc2ae585-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cd97f7fc7-54tcg\" (UID: \"22f3c1ce-3171-49cf-8a07-4c0fcc2ae585\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd97f7fc7-54tcg"
Jul 17 11:02:49.273853 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.273855    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zx7z\" (UniqueName: \"kubernetes.io/projected/22f3c1ce-3171-49cf-8a07-4c0fcc2ae585-kube-api-access-5zx7z\") pod \"opendatahub-operator-controller-manager-6cd97f7fc7-54tcg\" (UID: \"22f3c1ce-3171-49cf-8a07-4c0fcc2ae585\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd97f7fc7-54tcg"
Jul 17 11:02:49.274172 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.273890    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22f3c1ce-3171-49cf-8a07-4c0fcc2ae585-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cd97f7fc7-54tcg\" (UID: \"22f3c1ce-3171-49cf-8a07-4c0fcc2ae585\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd97f7fc7-54tcg"
Jul 17 11:02:49.274172 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.274006    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22f3c1ce-3171-49cf-8a07-4c0fcc2ae585-tmp\") pod \"opendatahub-operator-controller-manager-6cd97f7fc7-54tcg\" (UID: \"22f3c1ce-3171-49cf-8a07-4c0fcc2ae585\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd97f7fc7-54tcg"
Jul 17 11:02:49.274172 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.274053    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/22f3c1ce-3171-49cf-8a07-4c0fcc2ae585-manifests\") pod \"opendatahub-operator-controller-manager-6cd97f7fc7-54tcg\" (UID: \"22f3c1ce-3171-49cf-8a07-4c0fcc2ae585\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd97f7fc7-54tcg"
Jul 17 11:02:49.274482 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.274462    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/22f3c1ce-3171-49cf-8a07-4c0fcc2ae585-manifests\") pod \"opendatahub-operator-controller-manager-6cd97f7fc7-54tcg\" (UID: \"22f3c1ce-3171-49cf-8a07-4c0fcc2ae585\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd97f7fc7-54tcg"
Jul 17 11:02:49.276203 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.276167    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22f3c1ce-3171-49cf-8a07-4c0fcc2ae585-tmp\") pod \"opendatahub-operator-controller-manager-6cd97f7fc7-54tcg\" (UID: \"22f3c1ce-3171-49cf-8a07-4c0fcc2ae585\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd97f7fc7-54tcg"
Jul 17 11:02:49.276385 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.276369    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22f3c1ce-3171-49cf-8a07-4c0fcc2ae585-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cd97f7fc7-54tcg\" (UID: \"22f3c1ce-3171-49cf-8a07-4c0fcc2ae585\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd97f7fc7-54tcg"
Jul 17 11:02:49.276431 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.276409    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/22f3c1ce-3171-49cf-8a07-4c0fcc2ae585-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cd97f7fc7-54tcg\" (UID: \"22f3c1ce-3171-49cf-8a07-4c0fcc2ae585\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd97f7fc7-54tcg"
Jul 17 11:02:49.288903 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.288874    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zx7z\" (UniqueName: \"kubernetes.io/projected/22f3c1ce-3171-49cf-8a07-4c0fcc2ae585-kube-api-access-5zx7z\") pod \"opendatahub-operator-controller-manager-6cd97f7fc7-54tcg\" (UID: \"22f3c1ce-3171-49cf-8a07-4c0fcc2ae585\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd97f7fc7-54tcg"
Jul 17 11:02:49.329942 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.329903    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cd97f7fc7-54tcg"
Jul 17 11:02:49.349250 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.349217    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:49.450523 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.450453    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:49.474393 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.474230    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cd97f7fc7-54tcg"]
Jul 17 11:02:49.477309 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:02:49.477275    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f3c1ce_3171_49cf_8a07_4c0fcc2ae585.slice/crio-27f13495c485e6ca40e4a8d3d31f6806405921c23b9ff37b04cca0886b6fab4e WatchSource:0}: Error finding container 27f13495c485e6ca40e4a8d3d31f6806405921c23b9ff37b04cca0886b6fab4e: Status 404 returned error can't find the container with id 27f13495c485e6ca40e4a8d3d31f6806405921c23b9ff37b04cca0886b6fab4e
Jul 17 11:02:49.577775 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.577742    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:49.698528 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.698491    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:49.800119 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:49.800001    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:50.109699 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:50.109613    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cd97f7fc7-54tcg" event={"ID":"22f3c1ce-3171-49cf-8a07-4c0fcc2ae585","Type":"ContainerStarted","Data":"27f13495c485e6ca40e4a8d3d31f6806405921c23b9ff37b04cca0886b6fab4e"}
Jul 17 11:02:53.125845 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:53.125804    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cd97f7fc7-54tcg" event={"ID":"22f3c1ce-3171-49cf-8a07-4c0fcc2ae585","Type":"ContainerStarted","Data":"5ffaba76ff35bfa65f0410051502f443f07a3c978a90323ad7adc02a9340b7c3"}
Jul 17 11:02:54.132328 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:54.132277    2563 generic.go:356] "Generic (PLEG): container finished" podID="22f3c1ce-3171-49cf-8a07-4c0fcc2ae585" containerID="5ffaba76ff35bfa65f0410051502f443f07a3c978a90323ad7adc02a9340b7c3" exitCode=0
Jul 17 11:02:54.132796 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:54.132342    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cd97f7fc7-54tcg" event={"ID":"22f3c1ce-3171-49cf-8a07-4c0fcc2ae585","Type":"ContainerDied","Data":"5ffaba76ff35bfa65f0410051502f443f07a3c978a90323ad7adc02a9340b7c3"}
Jul 17 11:02:55.138315 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:55.138280    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cd97f7fc7-54tcg" event={"ID":"22f3c1ce-3171-49cf-8a07-4c0fcc2ae585","Type":"ContainerStarted","Data":"6ab1da505b3cee5477e0eb83c3ae4ed45c2adceb3d34e4b92596dbb98026b8ec"}
Jul 17 11:02:55.138696 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:55.138380    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cd97f7fc7-54tcg"
Jul 17 11:02:55.163552 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:55.163506    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6cd97f7fc7-54tcg" podStartSLOduration=2.49950479 podStartE2EDuration="7.16349024s" totalImagesPullingTime="3.62268815s" totalInitContainerRuntime="1.0412973s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:02:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:02:55.162199332 +0000 UTC m=+330.091738972" watchObservedRunningTime="2026-07-17 11:02:55.16349024 +0000 UTC m=+330.093029879"
Jul 17 11:02:57.249731 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:57.249698    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:02:57.345375 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:02:57.345346    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:03:06.145226 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:06.145188    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6cd97f7fc7-54tcg"
Jul 17 11:03:08.087563 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:08.087526    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:03:08.825900 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:08.825867    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-c556c8c74-kh79v"]
Jul 17 11:03:08.832414 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:08.832387    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-c556c8c74-kh79v"
Jul 17 11:03:08.835218 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:08.835190    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 17 11:03:08.835503 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:08.835480    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 17 11:03:08.836368 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:08.836353    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-24d95\""
Jul 17 11:03:08.836447 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:08.836411    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 17 11:03:08.844391 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:08.844369    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-c556c8c74-kh79v"]
Jul 17 11:03:08.845129 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:08.845113    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 17 11:03:08.942510 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:08.942474    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2fbn\" (UniqueName: \"kubernetes.io/projected/2a51e381-c349-4c91-b00c-460e28017f4a-kube-api-access-r2fbn\") pod \"kube-auth-proxy-c556c8c74-kh79v\" (UID: \"2a51e381-c349-4c91-b00c-460e28017f4a\") " pod="openshift-ingress/kube-auth-proxy-c556c8c74-kh79v"
Jul 17 11:03:08.942678 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:08.942585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a51e381-c349-4c91-b00c-460e28017f4a-tls-certs\") pod \"kube-auth-proxy-c556c8c74-kh79v\" (UID: \"2a51e381-c349-4c91-b00c-460e28017f4a\") " pod="openshift-ingress/kube-auth-proxy-c556c8c74-kh79v"
Jul 17 11:03:08.942678 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:08.942617    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a51e381-c349-4c91-b00c-460e28017f4a-tmp\") pod \"kube-auth-proxy-c556c8c74-kh79v\" (UID: \"2a51e381-c349-4c91-b00c-460e28017f4a\") " pod="openshift-ingress/kube-auth-proxy-c556c8c74-kh79v"
Jul 17 11:03:08.965293 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:08.965264    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rc9bz"]
Jul 17 11:03:08.967729 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:08.967698    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rc9bz"
Jul 17 11:03:08.977853 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:08.977830    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2v8ss\""
Jul 17 11:03:08.978477 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:08.978448    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 17 11:03:08.986431 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:08.986405    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rc9bz"]
Jul 17 11:03:09.043903 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:09.043874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2fbn\" (UniqueName: \"kubernetes.io/projected/2a51e381-c349-4c91-b00c-460e28017f4a-kube-api-access-r2fbn\") pod \"kube-auth-proxy-c556c8c74-kh79v\" (UID: \"2a51e381-c349-4c91-b00c-460e28017f4a\") " pod="openshift-ingress/kube-auth-proxy-c556c8c74-kh79v"
Jul 17 11:03:09.044084 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:09.043914    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ced4cbc-2492-41cd-b739-c8a54bade76b-cert\") pod \"odh-model-controller-7998c67dc4-rc9bz\" (UID: \"4ced4cbc-2492-41cd-b739-c8a54bade76b\") " pod="opendatahub/odh-model-controller-7998c67dc4-rc9bz"
Jul 17 11:03:09.044084 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:09.043939    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv7wn\" (UniqueName: \"kubernetes.io/projected/4ced4cbc-2492-41cd-b739-c8a54bade76b-kube-api-access-jv7wn\") pod \"odh-model-controller-7998c67dc4-rc9bz\" (UID: \"4ced4cbc-2492-41cd-b739-c8a54bade76b\") " pod="opendatahub/odh-model-controller-7998c67dc4-rc9bz"
Jul 17 11:03:09.044084 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:09.044002    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a51e381-c349-4c91-b00c-460e28017f4a-tls-certs\") pod \"kube-auth-proxy-c556c8c74-kh79v\" (UID: \"2a51e381-c349-4c91-b00c-460e28017f4a\") " pod="openshift-ingress/kube-auth-proxy-c556c8c74-kh79v"
Jul 17 11:03:09.044084 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:09.044024    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a51e381-c349-4c91-b00c-460e28017f4a-tmp\") pod \"kube-auth-proxy-c556c8c74-kh79v\" (UID: \"2a51e381-c349-4c91-b00c-460e28017f4a\") " pod="openshift-ingress/kube-auth-proxy-c556c8c74-kh79v"
Jul 17 11:03:09.046325 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:09.046296    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a51e381-c349-4c91-b00c-460e28017f4a-tmp\") pod \"kube-auth-proxy-c556c8c74-kh79v\" (UID: \"2a51e381-c349-4c91-b00c-460e28017f4a\") " pod="openshift-ingress/kube-auth-proxy-c556c8c74-kh79v"
Jul 17 11:03:09.046486 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:09.046471    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a51e381-c349-4c91-b00c-460e28017f4a-tls-certs\") pod \"kube-auth-proxy-c556c8c74-kh79v\" (UID: \"2a51e381-c349-4c91-b00c-460e28017f4a\") " pod="openshift-ingress/kube-auth-proxy-c556c8c74-kh79v"
Jul 17 11:03:09.061050 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:09.061024    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2fbn\" (UniqueName: \"kubernetes.io/projected/2a51e381-c349-4c91-b00c-460e28017f4a-kube-api-access-r2fbn\") pod \"kube-auth-proxy-c556c8c74-kh79v\" (UID: \"2a51e381-c349-4c91-b00c-460e28017f4a\") " pod="openshift-ingress/kube-auth-proxy-c556c8c74-kh79v"
Jul 17 11:03:09.141948 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:09.141917    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-c556c8c74-kh79v"
Jul 17 11:03:09.144663 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:09.144629    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ced4cbc-2492-41cd-b739-c8a54bade76b-cert\") pod \"odh-model-controller-7998c67dc4-rc9bz\" (UID: \"4ced4cbc-2492-41cd-b739-c8a54bade76b\") " pod="opendatahub/odh-model-controller-7998c67dc4-rc9bz"
Jul 17 11:03:09.144799 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:09.144679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv7wn\" (UniqueName: \"kubernetes.io/projected/4ced4cbc-2492-41cd-b739-c8a54bade76b-kube-api-access-jv7wn\") pod \"odh-model-controller-7998c67dc4-rc9bz\" (UID: \"4ced4cbc-2492-41cd-b739-c8a54bade76b\") " pod="opendatahub/odh-model-controller-7998c67dc4-rc9bz"
Jul 17 11:03:09.144799 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:03:09.144744    2563 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 17 11:03:09.145201 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:03:09.144822    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ced4cbc-2492-41cd-b739-c8a54bade76b-cert podName:4ced4cbc-2492-41cd-b739-c8a54bade76b nodeName:}" failed. No retries permitted until 2026-07-17 11:03:09.644801731 +0000 UTC m=+344.574341362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4ced4cbc-2492-41cd-b739-c8a54bade76b-cert") pod "odh-model-controller-7998c67dc4-rc9bz" (UID: "4ced4cbc-2492-41cd-b739-c8a54bade76b") : secret "odh-model-controller-webhook-cert" not found
Jul 17 11:03:09.160806 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:09.160774    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv7wn\" (UniqueName: \"kubernetes.io/projected/4ced4cbc-2492-41cd-b739-c8a54bade76b-kube-api-access-jv7wn\") pod \"odh-model-controller-7998c67dc4-rc9bz\" (UID: \"4ced4cbc-2492-41cd-b739-c8a54bade76b\") " pod="opendatahub/odh-model-controller-7998c67dc4-rc9bz"
Jul 17 11:03:09.283371 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:09.283336    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-c556c8c74-kh79v"]
Jul 17 11:03:09.286549 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:03:09.286524    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a51e381_c349_4c91_b00c_460e28017f4a.slice/crio-52920443b90b0bddf379d6d30a6a42aeca6730e1ed9a51e57ccd3a4aba096574 WatchSource:0}: Error finding container 52920443b90b0bddf379d6d30a6a42aeca6730e1ed9a51e57ccd3a4aba096574: Status 404 returned error can't find the container with id 52920443b90b0bddf379d6d30a6a42aeca6730e1ed9a51e57ccd3a4aba096574
Jul 17 11:03:09.421895 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:09.421812    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:03:09.633541 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:09.633505    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:03:09.647931 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:09.647901    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ced4cbc-2492-41cd-b739-c8a54bade76b-cert\") pod \"odh-model-controller-7998c67dc4-rc9bz\" (UID: \"4ced4cbc-2492-41cd-b739-c8a54bade76b\") " pod="opendatahub/odh-model-controller-7998c67dc4-rc9bz"
Jul 17 11:03:09.650211 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:09.650191    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ced4cbc-2492-41cd-b739-c8a54bade76b-cert\") pod \"odh-model-controller-7998c67dc4-rc9bz\" (UID: \"4ced4cbc-2492-41cd-b739-c8a54bade76b\") " pod="opendatahub/odh-model-controller-7998c67dc4-rc9bz"
Jul 17 11:03:09.849489 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:09.849381    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:03:09.880984 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:09.880952    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rc9bz"
Jul 17 11:03:10.072666 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:10.072625    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rc9bz"]
Jul 17 11:03:10.074686 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:03:10.074659    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ced4cbc_2492_41cd_b739_c8a54bade76b.slice/crio-d73b85546406830192264c7a25ebedd63d6ae149ae5f74711a2d5912b94d5675 WatchSource:0}: Error finding container d73b85546406830192264c7a25ebedd63d6ae149ae5f74711a2d5912b94d5675: Status 404 returned error can't find the container with id d73b85546406830192264c7a25ebedd63d6ae149ae5f74711a2d5912b94d5675
Jul 17 11:03:10.208516 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:10.208478    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rc9bz" event={"ID":"4ced4cbc-2492-41cd-b739-c8a54bade76b","Type":"ContainerStarted","Data":"d73b85546406830192264c7a25ebedd63d6ae149ae5f74711a2d5912b94d5675"}
Jul 17 11:03:10.209733 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:10.209682    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-c556c8c74-kh79v" event={"ID":"2a51e381-c349-4c91-b00c-460e28017f4a","Type":"ContainerStarted","Data":"52920443b90b0bddf379d6d30a6a42aeca6730e1ed9a51e57ccd3a4aba096574"}
Jul 17 11:03:10.240385 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:10.240339    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:03:10.588953 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:10.588881    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:03:10.671748 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:10.671714    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:03:11.838449 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:11.838385    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:03:13.994937 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:13.994906    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:03:14.064840 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.064796    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:03:14.186619 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.186587    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:03:14.232795 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.232752    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-c556c8c74-kh79v" event={"ID":"2a51e381-c349-4c91-b00c-460e28017f4a","Type":"ContainerStarted","Data":"96fb953300203ddc376741c1569bd899b2fac2a93f0c9a50b45d591b85f6afbd"}
Jul 17 11:03:14.234274 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.234246    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rc9bz" event={"ID":"4ced4cbc-2492-41cd-b739-c8a54bade76b","Type":"ContainerStarted","Data":"32e3d413b363fb6dc4ed301c48deb4aa82a0d2ab82b66623b4da237074ea4c87"}
Jul 17 11:03:14.234397 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.234382    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-rc9bz"
Jul 17 11:03:14.255977 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.255873    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-c556c8c74-kh79v" podStartSLOduration=2.147763839 podStartE2EDuration="6.255852879s" totalImagesPullingTime="4.10808904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:03:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:03:14.255315256 +0000 UTC m=+349.184854894" watchObservedRunningTime="2026-07-17 11:03:14.255852879 +0000 UTC m=+349.185392519"
Jul 17 11:03:14.277387 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.277311    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-rc9bz" podStartSLOduration=2.955247305 podStartE2EDuration="6.277291043s" totalImagesPullingTime="3.322043738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:03:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:03:14.276760605 +0000 UTC m=+349.206300248" watchObservedRunningTime="2026-07-17 11:03:14.277291043 +0000 UTC m=+349.206830684"
Jul 17 11:03:14.317641 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.317610    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:03:14.540262 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.540164    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"]
Jul 17 11:03:14.543651 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.543621    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.546260 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.546237    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 17 11:03:14.546356 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.546274    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-jknhs\""
Jul 17 11:03:14.556936 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.556791    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"]
Jul 17 11:03:14.576318 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.576284    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:03:14.590985 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.590959    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c6f4f1-bf17-4be7-b422-99093e48f316-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l7mvw\" (UID: \"d6c6f4f1-bf17-4be7-b422-99093e48f316\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.591121 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.591002    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6c6f4f1-bf17-4be7-b422-99093e48f316-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l7mvw\" (UID: \"d6c6f4f1-bf17-4be7-b422-99093e48f316\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.591121 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.591024    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c6f4f1-bf17-4be7-b422-99093e48f316-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l7mvw\" (UID: \"d6c6f4f1-bf17-4be7-b422-99093e48f316\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.591196 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.591116    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6c6f4f1-bf17-4be7-b422-99093e48f316-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l7mvw\" (UID: \"d6c6f4f1-bf17-4be7-b422-99093e48f316\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.591196 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.591162    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6c6f4f1-bf17-4be7-b422-99093e48f316-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l7mvw\" (UID: \"d6c6f4f1-bf17-4be7-b422-99093e48f316\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.591196 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.591187    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6c6f4f1-bf17-4be7-b422-99093e48f316-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l7mvw\" (UID: \"d6c6f4f1-bf17-4be7-b422-99093e48f316\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.591296 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.591208    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c6f4f1-bf17-4be7-b422-99093e48f316-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l7mvw\" (UID: \"d6c6f4f1-bf17-4be7-b422-99093e48f316\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.591339 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.591292    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6c6f4f1-bf17-4be7-b422-99093e48f316-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l7mvw\" (UID: \"d6c6f4f1-bf17-4be7-b422-99093e48f316\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.591407 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.591335    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvg99\" (UniqueName: \"kubernetes.io/projected/d6c6f4f1-bf17-4be7-b422-99093e48f316-kube-api-access-bvg99\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l7mvw\" (UID: \"d6c6f4f1-bf17-4be7-b422-99093e48f316\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.692455 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.692414    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6c6f4f1-bf17-4be7-b422-99093e48f316-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l7mvw\" (UID: \"d6c6f4f1-bf17-4be7-b422-99093e48f316\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.692455 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.692461    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6c6f4f1-bf17-4be7-b422-99093e48f316-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l7mvw\" (UID: \"d6c6f4f1-bf17-4be7-b422-99093e48f316\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.692688 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.692576    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c6f4f1-bf17-4be7-b422-99093e48f316-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l7mvw\" (UID: \"d6c6f4f1-bf17-4be7-b422-99093e48f316\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.692688 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.692635    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6c6f4f1-bf17-4be7-b422-99093e48f316-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l7mvw\" (UID: \"d6c6f4f1-bf17-4be7-b422-99093e48f316\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.692688 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.692657    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvg99\" (UniqueName: \"kubernetes.io/projected/d6c6f4f1-bf17-4be7-b422-99093e48f316-kube-api-access-bvg99\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l7mvw\" (UID: \"d6c6f4f1-bf17-4be7-b422-99093e48f316\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.692688 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.692686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c6f4f1-bf17-4be7-b422-99093e48f316-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l7mvw\" (UID: \"d6c6f4f1-bf17-4be7-b422-99093e48f316\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.692889 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.692718    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6c6f4f1-bf17-4be7-b422-99093e48f316-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l7mvw\" (UID: \"d6c6f4f1-bf17-4be7-b422-99093e48f316\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.692889 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.692746    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c6f4f1-bf17-4be7-b422-99093e48f316-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l7mvw\" (UID: \"d6c6f4f1-bf17-4be7-b422-99093e48f316\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.692999 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.692965    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c6f4f1-bf17-4be7-b422-99093e48f316-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l7mvw\" (UID: \"d6c6f4f1-bf17-4be7-b422-99093e48f316\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.693054 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.693017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6c6f4f1-bf17-4be7-b422-99093e48f316-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l7mvw\" (UID: \"d6c6f4f1-bf17-4be7-b422-99093e48f316\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.693150 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.693065    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c6f4f1-bf17-4be7-b422-99093e48f316-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l7mvw\" (UID: \"d6c6f4f1-bf17-4be7-b422-99093e48f316\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.693150 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.693129    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6c6f4f1-bf17-4be7-b422-99093e48f316-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l7mvw\" (UID: \"d6c6f4f1-bf17-4be7-b422-99093e48f316\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.693262 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.693174    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c6f4f1-bf17-4be7-b422-99093e48f316-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l7mvw\" (UID: \"d6c6f4f1-bf17-4be7-b422-99093e48f316\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.693788 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.693767    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6c6f4f1-bf17-4be7-b422-99093e48f316-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l7mvw\" (UID: \"d6c6f4f1-bf17-4be7-b422-99093e48f316\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.694751 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.694731    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6c6f4f1-bf17-4be7-b422-99093e48f316-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l7mvw\" (UID: \"d6c6f4f1-bf17-4be7-b422-99093e48f316\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.695225 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.695209    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6c6f4f1-bf17-4be7-b422-99093e48f316-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l7mvw\" (UID: \"d6c6f4f1-bf17-4be7-b422-99093e48f316\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.701937 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.701912    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6c6f4f1-bf17-4be7-b422-99093e48f316-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l7mvw\" (UID: \"d6c6f4f1-bf17-4be7-b422-99093e48f316\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.702193 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.702173    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvg99\" (UniqueName: \"kubernetes.io/projected/d6c6f4f1-bf17-4be7-b422-99093e48f316-kube-api-access-bvg99\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4l7mvw\" (UID: \"d6c6f4f1-bf17-4be7-b422-99093e48f316\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.860539 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.860447    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:14.997892 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:14.997866    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"]
Jul 17 11:03:14.999829 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:03:14.999803    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c6f4f1_bf17_4be7_b422_99093e48f316.slice/crio-7d81c3640b8551775021ca263ddb27176a3379124cc23b2c41de3bf41dac2efb WatchSource:0}: Error finding container 7d81c3640b8551775021ca263ddb27176a3379124cc23b2c41de3bf41dac2efb: Status 404 returned error can't find the container with id 7d81c3640b8551775021ca263ddb27176a3379124cc23b2c41de3bf41dac2efb
Jul 17 11:03:15.240148 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:15.240106    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw" event={"ID":"d6c6f4f1-bf17-4be7-b422-99093e48f316","Type":"ContainerStarted","Data":"7d81c3640b8551775021ca263ddb27176a3379124cc23b2c41de3bf41dac2efb"}
Jul 17 11:03:17.280605 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:17.280512    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-d4bfb48f4-wdkwr"]
Jul 17 11:03:17.288562 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:17.288536    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d4bfb48f4-wdkwr"
Jul 17 11:03:17.293279 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:17.293250    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-qs7rv\""
Jul 17 11:03:17.293409 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:17.293314    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 17 11:03:17.293409 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:17.293271    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 17 11:03:17.293502 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:17.293352    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 17 11:03:17.301063 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:17.301041    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d4bfb48f4-wdkwr"]
Jul 17 11:03:17.315809 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:17.315587    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ef1e11f2-4d04-426e-b5b7-7ece736a18f8-manager-config\") pod \"lws-controller-manager-d4bfb48f4-wdkwr\" (UID: \"ef1e11f2-4d04-426e-b5b7-7ece736a18f8\") " pod="openshift-lws-operator/lws-controller-manager-d4bfb48f4-wdkwr"
Jul 17 11:03:17.315809 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:17.315677    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ef1e11f2-4d04-426e-b5b7-7ece736a18f8-metrics-cert\") pod \"lws-controller-manager-d4bfb48f4-wdkwr\" (UID: \"ef1e11f2-4d04-426e-b5b7-7ece736a18f8\") " pod="openshift-lws-operator/lws-controller-manager-d4bfb48f4-wdkwr"
Jul 17 11:03:17.315809 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:17.315707    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef1e11f2-4d04-426e-b5b7-7ece736a18f8-cert\") pod \"lws-controller-manager-d4bfb48f4-wdkwr\" (UID: \"ef1e11f2-4d04-426e-b5b7-7ece736a18f8\") " pod="openshift-lws-operator/lws-controller-manager-d4bfb48f4-wdkwr"
Jul 17 11:03:17.315809 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:17.315744    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j57gb\" (UniqueName: \"kubernetes.io/projected/ef1e11f2-4d04-426e-b5b7-7ece736a18f8-kube-api-access-j57gb\") pod \"lws-controller-manager-d4bfb48f4-wdkwr\" (UID: \"ef1e11f2-4d04-426e-b5b7-7ece736a18f8\") " pod="openshift-lws-operator/lws-controller-manager-d4bfb48f4-wdkwr"
Jul 17 11:03:17.416820 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:17.416788    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ef1e11f2-4d04-426e-b5b7-7ece736a18f8-metrics-cert\") pod \"lws-controller-manager-d4bfb48f4-wdkwr\" (UID: \"ef1e11f2-4d04-426e-b5b7-7ece736a18f8\") " pod="openshift-lws-operator/lws-controller-manager-d4bfb48f4-wdkwr"
Jul 17 11:03:17.416964 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:17.416831    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef1e11f2-4d04-426e-b5b7-7ece736a18f8-cert\") pod \"lws-controller-manager-d4bfb48f4-wdkwr\" (UID: \"ef1e11f2-4d04-426e-b5b7-7ece736a18f8\") " pod="openshift-lws-operator/lws-controller-manager-d4bfb48f4-wdkwr"
Jul 17 11:03:17.416964 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:17.416871    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j57gb\" (UniqueName: \"kubernetes.io/projected/ef1e11f2-4d04-426e-b5b7-7ece736a18f8-kube-api-access-j57gb\") pod \"lws-controller-manager-d4bfb48f4-wdkwr\" (UID: \"ef1e11f2-4d04-426e-b5b7-7ece736a18f8\") " pod="openshift-lws-operator/lws-controller-manager-d4bfb48f4-wdkwr"
Jul 17 11:03:17.416964 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:17.416930    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ef1e11f2-4d04-426e-b5b7-7ece736a18f8-manager-config\") pod \"lws-controller-manager-d4bfb48f4-wdkwr\" (UID: \"ef1e11f2-4d04-426e-b5b7-7ece736a18f8\") " pod="openshift-lws-operator/lws-controller-manager-d4bfb48f4-wdkwr"
Jul 17 11:03:17.417779 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:17.417751    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ef1e11f2-4d04-426e-b5b7-7ece736a18f8-manager-config\") pod \"lws-controller-manager-d4bfb48f4-wdkwr\" (UID: \"ef1e11f2-4d04-426e-b5b7-7ece736a18f8\") " pod="openshift-lws-operator/lws-controller-manager-d4bfb48f4-wdkwr"
Jul 17 11:03:17.419312 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:17.419292    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef1e11f2-4d04-426e-b5b7-7ece736a18f8-cert\") pod \"lws-controller-manager-d4bfb48f4-wdkwr\" (UID: \"ef1e11f2-4d04-426e-b5b7-7ece736a18f8\") " pod="openshift-lws-operator/lws-controller-manager-d4bfb48f4-wdkwr"
Jul 17 11:03:17.419427 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:17.419356    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ef1e11f2-4d04-426e-b5b7-7ece736a18f8-metrics-cert\") pod \"lws-controller-manager-d4bfb48f4-wdkwr\" (UID: \"ef1e11f2-4d04-426e-b5b7-7ece736a18f8\") " pod="openshift-lws-operator/lws-controller-manager-d4bfb48f4-wdkwr"
Jul 17 11:03:17.428570 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:17.428545    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j57gb\" (UniqueName: \"kubernetes.io/projected/ef1e11f2-4d04-426e-b5b7-7ece736a18f8-kube-api-access-j57gb\") pod \"lws-controller-manager-d4bfb48f4-wdkwr\" (UID: \"ef1e11f2-4d04-426e-b5b7-7ece736a18f8\") " pod="openshift-lws-operator/lws-controller-manager-d4bfb48f4-wdkwr"
Jul 17 11:03:17.506506 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:17.506462    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 17 11:03:17.506607 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:17.506549    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 17 11:03:17.506607 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:17.506580    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 17 11:03:17.600697 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:17.600666    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d4bfb48f4-wdkwr"
Jul 17 11:03:17.770033 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:17.770002    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d4bfb48f4-wdkwr"]
Jul 17 11:03:17.773492 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:03:17.773464    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef1e11f2_4d04_426e_b5b7_7ece736a18f8.slice/crio-26711048d9d2076470490b279658953342e3b9374f36654e72c78df3daf83b93 WatchSource:0}: Error finding container 26711048d9d2076470490b279658953342e3b9374f36654e72c78df3daf83b93: Status 404 returned error can't find the container with id 26711048d9d2076470490b279658953342e3b9374f36654e72c78df3daf83b93
Jul 17 11:03:18.257431 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:18.257393    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw" event={"ID":"d6c6f4f1-bf17-4be7-b422-99093e48f316","Type":"ContainerStarted","Data":"d51f3027fef8886478db3c702cff52de517bc80fc0ef8dc0db838f35a59d88a3"}
Jul 17 11:03:18.258648 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:18.258619    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d4bfb48f4-wdkwr" event={"ID":"ef1e11f2-4d04-426e-b5b7-7ece736a18f8","Type":"ContainerStarted","Data":"26711048d9d2076470490b279658953342e3b9374f36654e72c78df3daf83b93"}
Jul 17 11:03:18.305342 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:18.305290    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw" podStartSLOduration=1.8010009 podStartE2EDuration="4.305274747s" totalImagesPullingTime="2.504273847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:03:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:03:18.304630516 +0000 UTC m=+353.234170157" watchObservedRunningTime="2026-07-17 11:03:18.305274747 +0000 UTC m=+353.234814387"
Jul 17 11:03:18.860861 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:18.860833    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:18.862150 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:18.862122    2563 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Jul 17 11:03:18.862277 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:18.862196    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw" podUID="d6c6f4f1-bf17-4be7-b422-99093e48f316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Jul 17 11:03:19.866008 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:19.865963    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:20.271002 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:20.270745    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:03:20.275655 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:20.275620    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d4bfb48f4-wdkwr" event={"ID":"ef1e11f2-4d04-426e-b5b7-7ece736a18f8","Type":"ContainerStarted","Data":"9a08cb59494c82d3de2315a7c4881513e10b4e33716cf8f0eb6cb51283926c31"}
Jul 17 11:03:20.275822 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:20.275710    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-d4bfb48f4-wdkwr"
Jul 17 11:03:20.275822 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:20.275748    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:20.276872 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:20.276852    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4l7mvw"
Jul 17 11:03:20.385475 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:20.385416    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-d4bfb48f4-wdkwr" podStartSLOduration=1.1262621720000001 podStartE2EDuration="3.385399744s" totalImagesPullingTime="2.259137572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:03:20.384107853 +0000 UTC m=+355.313647494" watchObservedRunningTime="2026-07-17 11:03:20.385399744 +0000 UTC m=+355.314939384"
Jul 17 11:03:20.479418 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:20.479388    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:03:20.593031 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:20.592945    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:03:20.766606 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:20.766573    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:03:21.724756 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:21.724720    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:03:21.850458 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:21.850413    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:03:25.244554 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:25.244525    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-rc9bz"
Jul 17 11:03:31.282177 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:03:31.282139    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-d4bfb48f4-wdkwr"
Jul 17 11:04:09.624867 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:09.624809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:04:09.719499 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:09.719470    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:04:10.807003 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:10.806969    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:04:24.956891 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:24.956842    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb"]
Jul 17 11:04:24.960182 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:24.960166    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb"
Jul 17 11:04:24.962707 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:24.962679    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 17 11:04:24.962815 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:24.962679    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 17 11:04:24.962815 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:24.962783    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5xfhp\""
Jul 17 11:04:24.969147 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:24.969105    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb"]
Jul 17 11:04:25.102489 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:25.102446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e09d1b3-7cdb-44b1-a080-5381ac1db7d5-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb\" (UID: \"7e09d1b3-7cdb-44b1-a080-5381ac1db7d5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb"
Jul 17 11:04:25.102677 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:25.102531    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e09d1b3-7cdb-44b1-a080-5381ac1db7d5-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb\" (UID: \"7e09d1b3-7cdb-44b1-a080-5381ac1db7d5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb"
Jul 17 11:04:25.102677 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:25.102559    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdnsh\" (UniqueName: \"kubernetes.io/projected/7e09d1b3-7cdb-44b1-a080-5381ac1db7d5-kube-api-access-vdnsh\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb\" (UID: \"7e09d1b3-7cdb-44b1-a080-5381ac1db7d5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb"
Jul 17 11:04:25.203783 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:25.203749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e09d1b3-7cdb-44b1-a080-5381ac1db7d5-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb\" (UID: \"7e09d1b3-7cdb-44b1-a080-5381ac1db7d5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb"
Jul 17 11:04:25.203976 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:25.203839    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e09d1b3-7cdb-44b1-a080-5381ac1db7d5-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb\" (UID: \"7e09d1b3-7cdb-44b1-a080-5381ac1db7d5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb"
Jul 17 11:04:25.203976 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:25.203868    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdnsh\" (UniqueName: \"kubernetes.io/projected/7e09d1b3-7cdb-44b1-a080-5381ac1db7d5-kube-api-access-vdnsh\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb\" (UID: \"7e09d1b3-7cdb-44b1-a080-5381ac1db7d5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb"
Jul 17 11:04:25.204241 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:25.204216    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e09d1b3-7cdb-44b1-a080-5381ac1db7d5-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb\" (UID: \"7e09d1b3-7cdb-44b1-a080-5381ac1db7d5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb"
Jul 17 11:04:25.204241 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:25.204235    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e09d1b3-7cdb-44b1-a080-5381ac1db7d5-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb\" (UID: \"7e09d1b3-7cdb-44b1-a080-5381ac1db7d5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb"
Jul 17 11:04:25.213683 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:25.213615    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdnsh\" (UniqueName: \"kubernetes.io/projected/7e09d1b3-7cdb-44b1-a080-5381ac1db7d5-kube-api-access-vdnsh\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb\" (UID: \"7e09d1b3-7cdb-44b1-a080-5381ac1db7d5\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb"
Jul 17 11:04:25.269536 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:25.269497    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb"
Jul 17 11:04:25.402803 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:25.402775    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb"]
Jul 17 11:04:25.405239 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:04:25.405186    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e09d1b3_7cdb_44b1_a080_5381ac1db7d5.slice/crio-d7554c9c7ec5ba20c94db8d6d2ae7f879e7d140c3af7497f168e97563704b838 WatchSource:0}: Error finding container d7554c9c7ec5ba20c94db8d6d2ae7f879e7d140c3af7497f168e97563704b838: Status 404 returned error can't find the container with id d7554c9c7ec5ba20c94db8d6d2ae7f879e7d140c3af7497f168e97563704b838
Jul 17 11:04:25.568907 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:25.568869    2563 generic.go:356] "Generic (PLEG): container finished" podID="7e09d1b3-7cdb-44b1-a080-5381ac1db7d5" containerID="2edbd1b76191f839d00499b7a45b1a62d538a7ea8afdb4b5e98e6fbda8419f1c" exitCode=0
Jul 17 11:04:25.569094 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:25.568915    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb" event={"ID":"7e09d1b3-7cdb-44b1-a080-5381ac1db7d5","Type":"ContainerDied","Data":"2edbd1b76191f839d00499b7a45b1a62d538a7ea8afdb4b5e98e6fbda8419f1c"}
Jul 17 11:04:25.569094 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:25.568968    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb" event={"ID":"7e09d1b3-7cdb-44b1-a080-5381ac1db7d5","Type":"ContainerStarted","Data":"d7554c9c7ec5ba20c94db8d6d2ae7f879e7d140c3af7497f168e97563704b838"}
Jul 17 11:04:25.749629 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:25.749593    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh"]
Jul 17 11:04:25.752908 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:25.752868    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh"
Jul 17 11:04:25.762797 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:25.762773    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh"]
Jul 17 11:04:25.809214 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:25.809173    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/229ead90-eb4a-4763-b2bd-757a963dea6b-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh\" (UID: \"229ead90-eb4a-4763-b2bd-757a963dea6b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh"
Jul 17 11:04:25.809452 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:25.809245    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7nd4\" (UniqueName: \"kubernetes.io/projected/229ead90-eb4a-4763-b2bd-757a963dea6b-kube-api-access-d7nd4\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh\" (UID: \"229ead90-eb4a-4763-b2bd-757a963dea6b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh"
Jul 17 11:04:25.809452 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:25.809279    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/229ead90-eb4a-4763-b2bd-757a963dea6b-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh\" (UID: \"229ead90-eb4a-4763-b2bd-757a963dea6b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh"
Jul 17 11:04:25.909766 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:25.909722    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7nd4\" (UniqueName: \"kubernetes.io/projected/229ead90-eb4a-4763-b2bd-757a963dea6b-kube-api-access-d7nd4\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh\" (UID: \"229ead90-eb4a-4763-b2bd-757a963dea6b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh"
Jul 17 11:04:25.909966 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:25.909777    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/229ead90-eb4a-4763-b2bd-757a963dea6b-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh\" (UID: \"229ead90-eb4a-4763-b2bd-757a963dea6b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh"
Jul 17 11:04:25.909966 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:25.909862    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/229ead90-eb4a-4763-b2bd-757a963dea6b-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh\" (UID: \"229ead90-eb4a-4763-b2bd-757a963dea6b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh"
Jul 17 11:04:25.910265 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:25.910244    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/229ead90-eb4a-4763-b2bd-757a963dea6b-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh\" (UID: \"229ead90-eb4a-4763-b2bd-757a963dea6b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh"
Jul 17 11:04:25.910345 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:25.910278    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/229ead90-eb4a-4763-b2bd-757a963dea6b-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh\" (UID: \"229ead90-eb4a-4763-b2bd-757a963dea6b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh"
Jul 17 11:04:25.923274 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:25.923245    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7nd4\" (UniqueName: \"kubernetes.io/projected/229ead90-eb4a-4763-b2bd-757a963dea6b-kube-api-access-d7nd4\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh\" (UID: \"229ead90-eb4a-4763-b2bd-757a963dea6b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh"
Jul 17 11:04:26.062610 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.062533    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh"
Jul 17 11:04:26.183943 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.183891    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls"]
Jul 17 11:04:26.188694 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.188674    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls"
Jul 17 11:04:26.197930 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.197899    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls"]
Jul 17 11:04:26.288098 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.288056    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh"]
Jul 17 11:04:26.290583 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:04:26.290555    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod229ead90_eb4a_4763_b2bd_757a963dea6b.slice/crio-f2ee657f511b44623f44b056b2894a9241745ee7c96887c704d36d78e70e5d37 WatchSource:0}: Error finding container f2ee657f511b44623f44b056b2894a9241745ee7c96887c704d36d78e70e5d37: Status 404 returned error can't find the container with id f2ee657f511b44623f44b056b2894a9241745ee7c96887c704d36d78e70e5d37
Jul 17 11:04:26.313236 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.313175    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d954e900-5ec9-46df-8d79-eaa33f8fdf5b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls\" (UID: \"d954e900-5ec9-46df-8d79-eaa33f8fdf5b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls"
Jul 17 11:04:26.313236 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.313224    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6657l\" (UniqueName: \"kubernetes.io/projected/d954e900-5ec9-46df-8d79-eaa33f8fdf5b-kube-api-access-6657l\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls\" (UID: \"d954e900-5ec9-46df-8d79-eaa33f8fdf5b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls"
Jul 17 11:04:26.313466 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.313323    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d954e900-5ec9-46df-8d79-eaa33f8fdf5b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls\" (UID: \"d954e900-5ec9-46df-8d79-eaa33f8fdf5b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls"
Jul 17 11:04:26.414639 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.414601    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d954e900-5ec9-46df-8d79-eaa33f8fdf5b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls\" (UID: \"d954e900-5ec9-46df-8d79-eaa33f8fdf5b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls"
Jul 17 11:04:26.414820 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.414650    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6657l\" (UniqueName: \"kubernetes.io/projected/d954e900-5ec9-46df-8d79-eaa33f8fdf5b-kube-api-access-6657l\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls\" (UID: \"d954e900-5ec9-46df-8d79-eaa33f8fdf5b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls"
Jul 17 11:04:26.414820 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.414702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d954e900-5ec9-46df-8d79-eaa33f8fdf5b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls\" (UID: \"d954e900-5ec9-46df-8d79-eaa33f8fdf5b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls"
Jul 17 11:04:26.415167 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.415149    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d954e900-5ec9-46df-8d79-eaa33f8fdf5b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls\" (UID: \"d954e900-5ec9-46df-8d79-eaa33f8fdf5b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls"
Jul 17 11:04:26.415211 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.415152    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d954e900-5ec9-46df-8d79-eaa33f8fdf5b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls\" (UID: \"d954e900-5ec9-46df-8d79-eaa33f8fdf5b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls"
Jul 17 11:04:26.427673 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.427644    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6657l\" (UniqueName: \"kubernetes.io/projected/d954e900-5ec9-46df-8d79-eaa33f8fdf5b-kube-api-access-6657l\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls\" (UID: \"d954e900-5ec9-46df-8d79-eaa33f8fdf5b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls"
Jul 17 11:04:26.505045 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.505014    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls"
Jul 17 11:04:26.551789 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.551758    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl"]
Jul 17 11:04:26.556061 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.556037    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl"
Jul 17 11:04:26.564993 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.564921    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl"]
Jul 17 11:04:26.576463 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.576435    2563 generic.go:356] "Generic (PLEG): container finished" podID="7e09d1b3-7cdb-44b1-a080-5381ac1db7d5" containerID="0710846d480cc133e50a7005da2b0cefa19da418bbe586e02d3c56623c900be7" exitCode=0
Jul 17 11:04:26.576624 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.576528    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb" event={"ID":"7e09d1b3-7cdb-44b1-a080-5381ac1db7d5","Type":"ContainerDied","Data":"0710846d480cc133e50a7005da2b0cefa19da418bbe586e02d3c56623c900be7"}
Jul 17 11:04:26.578398 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.578370    2563 generic.go:356] "Generic (PLEG): container finished" podID="229ead90-eb4a-4763-b2bd-757a963dea6b" containerID="5c6b01d74b089c1cbe7a9c5716b42672c281cfb07c806ad872ac85cd26321f28" exitCode=0
Jul 17 11:04:26.578494 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.578411    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh" event={"ID":"229ead90-eb4a-4763-b2bd-757a963dea6b","Type":"ContainerDied","Data":"5c6b01d74b089c1cbe7a9c5716b42672c281cfb07c806ad872ac85cd26321f28"}
Jul 17 11:04:26.578494 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.578452    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh" event={"ID":"229ead90-eb4a-4763-b2bd-757a963dea6b","Type":"ContainerStarted","Data":"f2ee657f511b44623f44b056b2894a9241745ee7c96887c704d36d78e70e5d37"}
Jul 17 11:04:26.616391 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.616347    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f38ed71a-8a3c-44b6-9564-78ae054df646-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl\" (UID: \"f38ed71a-8a3c-44b6-9564-78ae054df646\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl"
Jul 17 11:04:26.616557 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.616406    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f38ed71a-8a3c-44b6-9564-78ae054df646-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl\" (UID: \"f38ed71a-8a3c-44b6-9564-78ae054df646\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl"
Jul 17 11:04:26.616557 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.616469    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8gzn\" (UniqueName: \"kubernetes.io/projected/f38ed71a-8a3c-44b6-9564-78ae054df646-kube-api-access-p8gzn\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl\" (UID: \"f38ed71a-8a3c-44b6-9564-78ae054df646\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl"
Jul 17 11:04:26.646890 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.646865    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls"]
Jul 17 11:04:26.649010 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:04:26.648987    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd954e900_5ec9_46df_8d79_eaa33f8fdf5b.slice/crio-fae7a1fd9d02bcb338ab5888d81424903e3465d3c4b1fcee1d3cf4ba27cd5946 WatchSource:0}: Error finding container fae7a1fd9d02bcb338ab5888d81424903e3465d3c4b1fcee1d3cf4ba27cd5946: Status 404 returned error can't find the container with id fae7a1fd9d02bcb338ab5888d81424903e3465d3c4b1fcee1d3cf4ba27cd5946
Jul 17 11:04:26.717743 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.717720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8gzn\" (UniqueName: \"kubernetes.io/projected/f38ed71a-8a3c-44b6-9564-78ae054df646-kube-api-access-p8gzn\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl\" (UID: \"f38ed71a-8a3c-44b6-9564-78ae054df646\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl"
Jul 17 11:04:26.718518 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.718491    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f38ed71a-8a3c-44b6-9564-78ae054df646-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl\" (UID: \"f38ed71a-8a3c-44b6-9564-78ae054df646\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl"
Jul 17 11:04:26.718633 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.718581    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f38ed71a-8a3c-44b6-9564-78ae054df646-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl\" (UID: \"f38ed71a-8a3c-44b6-9564-78ae054df646\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl"
Jul 17 11:04:26.718821 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.718800    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f38ed71a-8a3c-44b6-9564-78ae054df646-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl\" (UID: \"f38ed71a-8a3c-44b6-9564-78ae054df646\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl"
Jul 17 11:04:26.718902 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.718888    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f38ed71a-8a3c-44b6-9564-78ae054df646-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl\" (UID: \"f38ed71a-8a3c-44b6-9564-78ae054df646\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl"
Jul 17 11:04:26.726977 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.726953    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8gzn\" (UniqueName: \"kubernetes.io/projected/f38ed71a-8a3c-44b6-9564-78ae054df646-kube-api-access-p8gzn\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl\" (UID: \"f38ed71a-8a3c-44b6-9564-78ae054df646\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl"
Jul 17 11:04:26.870287 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:26.870200    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl"
Jul 17 11:04:27.008404 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:27.008380    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl"]
Jul 17 11:04:27.010105 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:04:27.010061    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf38ed71a_8a3c_44b6_9564_78ae054df646.slice/crio-a682b30809f9b7e4847c7a9b07c51e774f3d32e9d264c68e3662a01d41d7d241 WatchSource:0}: Error finding container a682b30809f9b7e4847c7a9b07c51e774f3d32e9d264c68e3662a01d41d7d241: Status 404 returned error can't find the container with id a682b30809f9b7e4847c7a9b07c51e774f3d32e9d264c68e3662a01d41d7d241
Jul 17 11:04:27.584748 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:27.584656    2563 generic.go:356] "Generic (PLEG): container finished" podID="f38ed71a-8a3c-44b6-9564-78ae054df646" containerID="581b73ed350eec465c77279f9511b8eccded5d69fe6e822bb367e1eb0a19d9c7" exitCode=0
Jul 17 11:04:27.585216 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:27.584746    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl" event={"ID":"f38ed71a-8a3c-44b6-9564-78ae054df646","Type":"ContainerDied","Data":"581b73ed350eec465c77279f9511b8eccded5d69fe6e822bb367e1eb0a19d9c7"}
Jul 17 11:04:27.585216 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:27.584808    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl" event={"ID":"f38ed71a-8a3c-44b6-9564-78ae054df646","Type":"ContainerStarted","Data":"a682b30809f9b7e4847c7a9b07c51e774f3d32e9d264c68e3662a01d41d7d241"}
Jul 17 11:04:27.586694 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:27.586666    2563 generic.go:356] "Generic (PLEG): container finished" podID="229ead90-eb4a-4763-b2bd-757a963dea6b" containerID="a631990821589e7cd203c11efe5d9d1e6a7ff78dc2f81dcc7eda2dec8a43b2a5" exitCode=0
Jul 17 11:04:27.586800 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:27.586721    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh" event={"ID":"229ead90-eb4a-4763-b2bd-757a963dea6b","Type":"ContainerDied","Data":"a631990821589e7cd203c11efe5d9d1e6a7ff78dc2f81dcc7eda2dec8a43b2a5"}
Jul 17 11:04:27.588867 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:27.588843    2563 generic.go:356] "Generic (PLEG): container finished" podID="7e09d1b3-7cdb-44b1-a080-5381ac1db7d5" containerID="de41482a40a0cb0f23cf3725311813234bd6ae51d5a9eaa0a27bb6cd016d97a6" exitCode=0
Jul 17 11:04:27.589037 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:27.588929    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb" event={"ID":"7e09d1b3-7cdb-44b1-a080-5381ac1db7d5","Type":"ContainerDied","Data":"de41482a40a0cb0f23cf3725311813234bd6ae51d5a9eaa0a27bb6cd016d97a6"}
Jul 17 11:04:27.590376 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:27.590346    2563 generic.go:356] "Generic (PLEG): container finished" podID="d954e900-5ec9-46df-8d79-eaa33f8fdf5b" containerID="2b72b4a25cfba264f3abb03a7de47fbf211dd559fa54a54db3f93f003f59eded" exitCode=0
Jul 17 11:04:27.590456 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:27.590380    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls" event={"ID":"d954e900-5ec9-46df-8d79-eaa33f8fdf5b","Type":"ContainerDied","Data":"2b72b4a25cfba264f3abb03a7de47fbf211dd559fa54a54db3f93f003f59eded"}
Jul 17 11:04:27.590456 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:27.590413    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls" event={"ID":"d954e900-5ec9-46df-8d79-eaa33f8fdf5b","Type":"ContainerStarted","Data":"fae7a1fd9d02bcb338ab5888d81424903e3465d3c4b1fcee1d3cf4ba27cd5946"}
Jul 17 11:04:28.596451 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:28.596358    2563 generic.go:356] "Generic (PLEG): container finished" podID="f38ed71a-8a3c-44b6-9564-78ae054df646" containerID="f46e1bc7649dccd40217cc46b54025b20503157999b3b534f8e7d3232989471d" exitCode=0
Jul 17 11:04:28.596858 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:28.596448    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl" event={"ID":"f38ed71a-8a3c-44b6-9564-78ae054df646","Type":"ContainerDied","Data":"f46e1bc7649dccd40217cc46b54025b20503157999b3b534f8e7d3232989471d"}
Jul 17 11:04:28.598654 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:28.598629    2563 generic.go:356] "Generic (PLEG): container finished" podID="229ead90-eb4a-4763-b2bd-757a963dea6b" containerID="b0d0d18b8fa84213b4dc8ffd159b6adc8b62cbeee1027a0d4297d74961787c6b" exitCode=0
Jul 17 11:04:28.598766 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:28.598746    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh" event={"ID":"229ead90-eb4a-4763-b2bd-757a963dea6b","Type":"ContainerDied","Data":"b0d0d18b8fa84213b4dc8ffd159b6adc8b62cbeee1027a0d4297d74961787c6b"}
Jul 17 11:04:28.797160 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:28.797134    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb"
Jul 17 11:04:28.939673 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:28.939638    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdnsh\" (UniqueName: \"kubernetes.io/projected/7e09d1b3-7cdb-44b1-a080-5381ac1db7d5-kube-api-access-vdnsh\") pod \"7e09d1b3-7cdb-44b1-a080-5381ac1db7d5\" (UID: \"7e09d1b3-7cdb-44b1-a080-5381ac1db7d5\") "
Jul 17 11:04:28.939824 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:28.939696    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e09d1b3-7cdb-44b1-a080-5381ac1db7d5-bundle\") pod \"7e09d1b3-7cdb-44b1-a080-5381ac1db7d5\" (UID: \"7e09d1b3-7cdb-44b1-a080-5381ac1db7d5\") "
Jul 17 11:04:28.939824 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:28.939768    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e09d1b3-7cdb-44b1-a080-5381ac1db7d5-util\") pod \"7e09d1b3-7cdb-44b1-a080-5381ac1db7d5\" (UID: \"7e09d1b3-7cdb-44b1-a080-5381ac1db7d5\") "
Jul 17 11:04:28.940237 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:28.940211    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e09d1b3-7cdb-44b1-a080-5381ac1db7d5-bundle" (OuterVolumeSpecName: "bundle") pod "7e09d1b3-7cdb-44b1-a080-5381ac1db7d5" (UID: "7e09d1b3-7cdb-44b1-a080-5381ac1db7d5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:04:28.941905 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:28.941885    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e09d1b3-7cdb-44b1-a080-5381ac1db7d5-kube-api-access-vdnsh" (OuterVolumeSpecName: "kube-api-access-vdnsh") pod "7e09d1b3-7cdb-44b1-a080-5381ac1db7d5" (UID: "7e09d1b3-7cdb-44b1-a080-5381ac1db7d5"). InnerVolumeSpecName "kube-api-access-vdnsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 11:04:28.945201 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:28.945166    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e09d1b3-7cdb-44b1-a080-5381ac1db7d5-util" (OuterVolumeSpecName: "util") pod "7e09d1b3-7cdb-44b1-a080-5381ac1db7d5" (UID: "7e09d1b3-7cdb-44b1-a080-5381ac1db7d5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:04:29.041369 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:29.041330    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdnsh\" (UniqueName: \"kubernetes.io/projected/7e09d1b3-7cdb-44b1-a080-5381ac1db7d5-kube-api-access-vdnsh\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:04:29.041369 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:29.041368    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e09d1b3-7cdb-44b1-a080-5381ac1db7d5-bundle\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:04:29.041647 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:29.041383    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e09d1b3-7cdb-44b1-a080-5381ac1db7d5-util\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:04:29.606367 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:29.606286    2563 generic.go:356] "Generic (PLEG): container finished" podID="f38ed71a-8a3c-44b6-9564-78ae054df646" containerID="1ec483920d61e3e61ca9df9264eab7275c0bc064200f868c3ea0cc27cf32117d" exitCode=0
Jul 17 11:04:29.606787 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:29.606367    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl" event={"ID":"f38ed71a-8a3c-44b6-9564-78ae054df646","Type":"ContainerDied","Data":"1ec483920d61e3e61ca9df9264eab7275c0bc064200f868c3ea0cc27cf32117d"}
Jul 17 11:04:29.608181 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:29.608163    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb"
Jul 17 11:04:29.608283 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:29.608159    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb" event={"ID":"7e09d1b3-7cdb-44b1-a080-5381ac1db7d5","Type":"ContainerDied","Data":"d7554c9c7ec5ba20c94db8d6d2ae7f879e7d140c3af7497f168e97563704b838"}
Jul 17 11:04:29.608283 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:29.608270    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7554c9c7ec5ba20c94db8d6d2ae7f879e7d140c3af7497f168e97563704b838"
Jul 17 11:04:29.609732 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:29.609709    2563 generic.go:356] "Generic (PLEG): container finished" podID="d954e900-5ec9-46df-8d79-eaa33f8fdf5b" containerID="5faa84d75742fe4cd6310ea9d1936ef74340339666ba71f23975086f9fa09861" exitCode=0
Jul 17 11:04:29.609830 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:29.609756    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls" event={"ID":"d954e900-5ec9-46df-8d79-eaa33f8fdf5b","Type":"ContainerDied","Data":"5faa84d75742fe4cd6310ea9d1936ef74340339666ba71f23975086f9fa09861"}
Jul 17 11:04:29.745283 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:29.745260    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh"
Jul 17 11:04:29.847194 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:29.847157    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7nd4\" (UniqueName: \"kubernetes.io/projected/229ead90-eb4a-4763-b2bd-757a963dea6b-kube-api-access-d7nd4\") pod \"229ead90-eb4a-4763-b2bd-757a963dea6b\" (UID: \"229ead90-eb4a-4763-b2bd-757a963dea6b\") "
Jul 17 11:04:29.847380 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:29.847263    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/229ead90-eb4a-4763-b2bd-757a963dea6b-bundle\") pod \"229ead90-eb4a-4763-b2bd-757a963dea6b\" (UID: \"229ead90-eb4a-4763-b2bd-757a963dea6b\") "
Jul 17 11:04:29.847380 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:29.847339    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/229ead90-eb4a-4763-b2bd-757a963dea6b-util\") pod \"229ead90-eb4a-4763-b2bd-757a963dea6b\" (UID: \"229ead90-eb4a-4763-b2bd-757a963dea6b\") "
Jul 17 11:04:29.847818 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:29.847785    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229ead90-eb4a-4763-b2bd-757a963dea6b-bundle" (OuterVolumeSpecName: "bundle") pod "229ead90-eb4a-4763-b2bd-757a963dea6b" (UID: "229ead90-eb4a-4763-b2bd-757a963dea6b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:04:29.849328 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:29.849304    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/229ead90-eb4a-4763-b2bd-757a963dea6b-kube-api-access-d7nd4" (OuterVolumeSpecName: "kube-api-access-d7nd4") pod "229ead90-eb4a-4763-b2bd-757a963dea6b" (UID: "229ead90-eb4a-4763-b2bd-757a963dea6b"). InnerVolumeSpecName "kube-api-access-d7nd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 11:04:29.854015 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:29.853980    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229ead90-eb4a-4763-b2bd-757a963dea6b-util" (OuterVolumeSpecName: "util") pod "229ead90-eb4a-4763-b2bd-757a963dea6b" (UID: "229ead90-eb4a-4763-b2bd-757a963dea6b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:04:29.948049 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:29.948010    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7nd4\" (UniqueName: \"kubernetes.io/projected/229ead90-eb4a-4763-b2bd-757a963dea6b-kube-api-access-d7nd4\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:04:29.948049 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:29.948043    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/229ead90-eb4a-4763-b2bd-757a963dea6b-bundle\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:04:29.948049 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:29.948054    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/229ead90-eb4a-4763-b2bd-757a963dea6b-util\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:04:30.616892 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:30.616818    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh"
Jul 17 11:04:30.617280 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:30.616810    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh" event={"ID":"229ead90-eb4a-4763-b2bd-757a963dea6b","Type":"ContainerDied","Data":"f2ee657f511b44623f44b056b2894a9241745ee7c96887c704d36d78e70e5d37"}
Jul 17 11:04:30.617280 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:30.616989    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2ee657f511b44623f44b056b2894a9241745ee7c96887c704d36d78e70e5d37"
Jul 17 11:04:30.618891 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:30.618868    2563 generic.go:356] "Generic (PLEG): container finished" podID="d954e900-5ec9-46df-8d79-eaa33f8fdf5b" containerID="3ee27409ec48862c406d0e14375e3de7c90620f733ea214950fe9851ab7bc6f2" exitCode=0
Jul 17 11:04:30.619006 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:30.618958    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls" event={"ID":"d954e900-5ec9-46df-8d79-eaa33f8fdf5b","Type":"ContainerDied","Data":"3ee27409ec48862c406d0e14375e3de7c90620f733ea214950fe9851ab7bc6f2"}
Jul 17 11:04:30.750556 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:30.750532    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl"
Jul 17 11:04:30.855403 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:30.855369    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8gzn\" (UniqueName: \"kubernetes.io/projected/f38ed71a-8a3c-44b6-9564-78ae054df646-kube-api-access-p8gzn\") pod \"f38ed71a-8a3c-44b6-9564-78ae054df646\" (UID: \"f38ed71a-8a3c-44b6-9564-78ae054df646\") "
Jul 17 11:04:30.855550 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:30.855441    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f38ed71a-8a3c-44b6-9564-78ae054df646-bundle\") pod \"f38ed71a-8a3c-44b6-9564-78ae054df646\" (UID: \"f38ed71a-8a3c-44b6-9564-78ae054df646\") "
Jul 17 11:04:30.855642 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:30.855618    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f38ed71a-8a3c-44b6-9564-78ae054df646-util\") pod \"f38ed71a-8a3c-44b6-9564-78ae054df646\" (UID: \"f38ed71a-8a3c-44b6-9564-78ae054df646\") "
Jul 17 11:04:30.855936 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:30.855906    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38ed71a-8a3c-44b6-9564-78ae054df646-bundle" (OuterVolumeSpecName: "bundle") pod "f38ed71a-8a3c-44b6-9564-78ae054df646" (UID: "f38ed71a-8a3c-44b6-9564-78ae054df646"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:04:30.857516 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:30.857496    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f38ed71a-8a3c-44b6-9564-78ae054df646-kube-api-access-p8gzn" (OuterVolumeSpecName: "kube-api-access-p8gzn") pod "f38ed71a-8a3c-44b6-9564-78ae054df646" (UID: "f38ed71a-8a3c-44b6-9564-78ae054df646"). InnerVolumeSpecName "kube-api-access-p8gzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 11:04:30.860787 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:30.860749    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38ed71a-8a3c-44b6-9564-78ae054df646-util" (OuterVolumeSpecName: "util") pod "f38ed71a-8a3c-44b6-9564-78ae054df646" (UID: "f38ed71a-8a3c-44b6-9564-78ae054df646"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:04:30.956834 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:30.956799    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8gzn\" (UniqueName: \"kubernetes.io/projected/f38ed71a-8a3c-44b6-9564-78ae054df646-kube-api-access-p8gzn\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:04:30.956834 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:30.956829    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f38ed71a-8a3c-44b6-9564-78ae054df646-bundle\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:04:30.956834 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:30.956838    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f38ed71a-8a3c-44b6-9564-78ae054df646-util\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:04:31.627488 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:31.627448    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl" event={"ID":"f38ed71a-8a3c-44b6-9564-78ae054df646","Type":"ContainerDied","Data":"a682b30809f9b7e4847c7a9b07c51e774f3d32e9d264c68e3662a01d41d7d241"}
Jul 17 11:04:31.627488 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:31.627479    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl"
Jul 17 11:04:31.627488 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:31.627496    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a682b30809f9b7e4847c7a9b07c51e774f3d32e9d264c68e3662a01d41d7d241"
Jul 17 11:04:31.768803 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:31.768780    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls"
Jul 17 11:04:31.865372 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:31.865342    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d954e900-5ec9-46df-8d79-eaa33f8fdf5b-bundle\") pod \"d954e900-5ec9-46df-8d79-eaa33f8fdf5b\" (UID: \"d954e900-5ec9-46df-8d79-eaa33f8fdf5b\") "
Jul 17 11:04:31.865528 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:31.865494    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6657l\" (UniqueName: \"kubernetes.io/projected/d954e900-5ec9-46df-8d79-eaa33f8fdf5b-kube-api-access-6657l\") pod \"d954e900-5ec9-46df-8d79-eaa33f8fdf5b\" (UID: \"d954e900-5ec9-46df-8d79-eaa33f8fdf5b\") "
Jul 17 11:04:31.865528 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:31.865523    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d954e900-5ec9-46df-8d79-eaa33f8fdf5b-util\") pod \"d954e900-5ec9-46df-8d79-eaa33f8fdf5b\" (UID: \"d954e900-5ec9-46df-8d79-eaa33f8fdf5b\") "
Jul 17 11:04:31.866103 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:31.866031    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d954e900-5ec9-46df-8d79-eaa33f8fdf5b-bundle" (OuterVolumeSpecName: "bundle") pod "d954e900-5ec9-46df-8d79-eaa33f8fdf5b" (UID: "d954e900-5ec9-46df-8d79-eaa33f8fdf5b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:04:31.867632 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:31.867608    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d954e900-5ec9-46df-8d79-eaa33f8fdf5b-kube-api-access-6657l" (OuterVolumeSpecName: "kube-api-access-6657l") pod "d954e900-5ec9-46df-8d79-eaa33f8fdf5b" (UID: "d954e900-5ec9-46df-8d79-eaa33f8fdf5b"). InnerVolumeSpecName "kube-api-access-6657l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 11:04:31.966906 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:31.966867    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6657l\" (UniqueName: \"kubernetes.io/projected/d954e900-5ec9-46df-8d79-eaa33f8fdf5b-kube-api-access-6657l\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:04:31.966906 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:31.966896    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d954e900-5ec9-46df-8d79-eaa33f8fdf5b-bundle\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:04:31.970315 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:31.970287    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d954e900-5ec9-46df-8d79-eaa33f8fdf5b-util" (OuterVolumeSpecName: "util") pod "d954e900-5ec9-46df-8d79-eaa33f8fdf5b" (UID: "d954e900-5ec9-46df-8d79-eaa33f8fdf5b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:04:32.068425 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:32.068391    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d954e900-5ec9-46df-8d79-eaa33f8fdf5b-util\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:04:32.639989 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:32.639950    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls" event={"ID":"d954e900-5ec9-46df-8d79-eaa33f8fdf5b","Type":"ContainerDied","Data":"fae7a1fd9d02bcb338ab5888d81424903e3465d3c4b1fcee1d3cf4ba27cd5946"}
Jul 17 11:04:32.639989 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:32.639994    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls"
Jul 17 11:04:32.640398 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:32.640037    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fae7a1fd9d02bcb338ab5888d81424903e3465d3c4b1fcee1d3cf4ba27cd5946"
Jul 17 11:04:37.984577 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:37.984547    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:04:38.581504 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:38.581465    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:04:40.188896 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:40.188861    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:04:40.291147 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:40.291111    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:04:42.796645 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:42.796617    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:04:42.938292 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:42.938256    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:04:47.597667 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:47.597588    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:04:47.696921 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:47.696887    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:04:49.007723 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:49.007689    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:04:49.229242 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:49.229206    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:04:51.989520 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:51.989485    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:04:52.049493 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:52.049454    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:04:52.404052 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:52.404021    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:04:52.477782 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:52.477746    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:04:54.036998 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:54.036963    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:04:54.144810 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:54.144778    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:04:54.240912 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:54.240880    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:04:54.767887 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:54.767855    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:04:55.861945 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:55.861898    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:04:57.285627 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:57.285596    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:04:57.427552 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:57.427520    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:04:57.534623 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:04:57.534590    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:05:04.414887 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:05:04.414848    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:05:04.741778 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:05:04.741698    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:05:04.841480 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:05:04.841454    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:05:05.144669 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:05:05.144639    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:05:05.246735 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:05:05.246699    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:05:06.963208 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:05:06.963170    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:05:07.061162 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:05:07.061129    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:05:08.890756 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:05:08.890721    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:05:09.147257 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:05:09.147173    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:05:12.618882 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:05:12.618852    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:05:12.720259 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:05:12.720223    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:05:13.248758 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:05:13.248723    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:05:13.325739 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:05:13.325705    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:05:17.242494 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:05:17.242459    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:05:17.312289 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:05:17.312259    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:05:25.038585 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:05:25.038547    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:05:25.147358 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:05:25.147320    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:05:28.637672 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:05:28.637638    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:05:28.734195 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:05:28.734155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:05:48.396437 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:05:48.396401    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:05:48.480966 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:05:48.480936    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:06:04.973535 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:04.973501    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:06:05.093306 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.093274    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:06:05.178407 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.178373    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:06:05.863318 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863282    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-9w45x"]
Jul 17 11:06:05.863611 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863599    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e09d1b3-7cdb-44b1-a080-5381ac1db7d5" containerName="pull"
Jul 17 11:06:05.863654 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863612    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e09d1b3-7cdb-44b1-a080-5381ac1db7d5" containerName="pull"
Jul 17 11:06:05.863654 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863621    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f38ed71a-8a3c-44b6-9564-78ae054df646" containerName="pull"
Jul 17 11:06:05.863654 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863626    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f38ed71a-8a3c-44b6-9564-78ae054df646" containerName="pull"
Jul 17 11:06:05.863654 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863632    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d954e900-5ec9-46df-8d79-eaa33f8fdf5b" containerName="extract"
Jul 17 11:06:05.863654 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863638    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d954e900-5ec9-46df-8d79-eaa33f8fdf5b" containerName="extract"
Jul 17 11:06:05.863654 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863645    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229ead90-eb4a-4763-b2bd-757a963dea6b" containerName="util"
Jul 17 11:06:05.863654 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863650    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="229ead90-eb4a-4763-b2bd-757a963dea6b" containerName="util"
Jul 17 11:06:05.863654 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863656    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229ead90-eb4a-4763-b2bd-757a963dea6b" containerName="extract"
Jul 17 11:06:05.863879 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863661    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="229ead90-eb4a-4763-b2bd-757a963dea6b" containerName="extract"
Jul 17 11:06:05.863879 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863670    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229ead90-eb4a-4763-b2bd-757a963dea6b" containerName="pull"
Jul 17 11:06:05.863879 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863675    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="229ead90-eb4a-4763-b2bd-757a963dea6b" containerName="pull"
Jul 17 11:06:05.863879 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863685    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f38ed71a-8a3c-44b6-9564-78ae054df646" containerName="extract"
Jul 17 11:06:05.863879 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863690    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f38ed71a-8a3c-44b6-9564-78ae054df646" containerName="extract"
Jul 17 11:06:05.863879 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863696    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e09d1b3-7cdb-44b1-a080-5381ac1db7d5" containerName="extract"
Jul 17 11:06:05.863879 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863701    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e09d1b3-7cdb-44b1-a080-5381ac1db7d5" containerName="extract"
Jul 17 11:06:05.863879 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863709    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d954e900-5ec9-46df-8d79-eaa33f8fdf5b" containerName="pull"
Jul 17 11:06:05.863879 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863714    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d954e900-5ec9-46df-8d79-eaa33f8fdf5b" containerName="pull"
Jul 17 11:06:05.863879 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863720    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e09d1b3-7cdb-44b1-a080-5381ac1db7d5" containerName="util"
Jul 17 11:06:05.863879 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863724    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e09d1b3-7cdb-44b1-a080-5381ac1db7d5" containerName="util"
Jul 17 11:06:05.863879 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863729    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f38ed71a-8a3c-44b6-9564-78ae054df646" containerName="util"
Jul 17 11:06:05.863879 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863734    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f38ed71a-8a3c-44b6-9564-78ae054df646" containerName="util"
Jul 17 11:06:05.863879 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863740    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d954e900-5ec9-46df-8d79-eaa33f8fdf5b" containerName="util"
Jul 17 11:06:05.863879 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863744    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d954e900-5ec9-46df-8d79-eaa33f8fdf5b" containerName="util"
Jul 17 11:06:05.863879 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863791    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d954e900-5ec9-46df-8d79-eaa33f8fdf5b" containerName="extract"
Jul 17 11:06:05.863879 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863800    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="229ead90-eb4a-4763-b2bd-757a963dea6b" containerName="extract"
Jul 17 11:06:05.863879 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863806    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f38ed71a-8a3c-44b6-9564-78ae054df646" containerName="extract"
Jul 17 11:06:05.863879 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.863813    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e09d1b3-7cdb-44b1-a080-5381ac1db7d5" containerName="extract"
Jul 17 11:06:05.866753 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.866730    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9w45x"
Jul 17 11:06:05.870478 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.870454    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 17 11:06:05.871325 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.871307    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qm856\""
Jul 17 11:06:05.872010 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.871990    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 17 11:06:05.877709 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.877685    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-9w45x"]
Jul 17 11:06:05.922903 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.922875    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:06:05.950155 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:05.950128    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5mhg\" (UniqueName: \"kubernetes.io/projected/2dbbe88d-b695-481e-91b7-8853c333c8b3-kube-api-access-g5mhg\") pod \"authorino-58cd955675-9w45x\" (UID: \"2dbbe88d-b695-481e-91b7-8853c333c8b3\") " pod="kuadrant-system/authorino-58cd955675-9w45x"
Jul 17 11:06:06.020495 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:06.020463    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:06:06.050899 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:06.050867    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5mhg\" (UniqueName: \"kubernetes.io/projected/2dbbe88d-b695-481e-91b7-8853c333c8b3-kube-api-access-g5mhg\") pod \"authorino-58cd955675-9w45x\" (UID: \"2dbbe88d-b695-481e-91b7-8853c333c8b3\") " pod="kuadrant-system/authorino-58cd955675-9w45x"
Jul 17 11:06:06.067126 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:06.067093    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5mhg\" (UniqueName: \"kubernetes.io/projected/2dbbe88d-b695-481e-91b7-8853c333c8b3-kube-api-access-g5mhg\") pod \"authorino-58cd955675-9w45x\" (UID: \"2dbbe88d-b695-481e-91b7-8853c333c8b3\") " pod="kuadrant-system/authorino-58cd955675-9w45x"
Jul 17 11:06:06.176051 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:06.176014    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9w45x"
Jul 17 11:06:06.309855 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:06.309818    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-9w45x"]
Jul 17 11:06:06.312118 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:06:06.312090    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dbbe88d_b695_481e_91b7_8853c333c8b3.slice/crio-780e60430a8e014924caa66a72bad97a4f56c0854dc075e66d187d9990b15c13 WatchSource:0}: Error finding container 780e60430a8e014924caa66a72bad97a4f56c0854dc075e66d187d9990b15c13: Status 404 returned error can't find the container with id 780e60430a8e014924caa66a72bad97a4f56c0854dc075e66d187d9990b15c13
Jul 17 11:06:07.087214 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:07.087158    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9w45x" event={"ID":"2dbbe88d-b695-481e-91b7-8853c333c8b3","Type":"ContainerStarted","Data":"780e60430a8e014924caa66a72bad97a4f56c0854dc075e66d187d9990b15c13"}
Jul 17 11:06:10.104420 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:10.104382    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9w45x" event={"ID":"2dbbe88d-b695-481e-91b7-8853c333c8b3","Type":"ContainerStarted","Data":"36aba5d1fce23b3f916bfc252617c0c7a5023ddf4c7dad47ebecf07569e0bd33"}
Jul 17 11:06:10.124619 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:10.124569    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-9w45x" podStartSLOduration=2.165604995 podStartE2EDuration="5.124553411s" totalImagesPullingTime="2.958948416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:06:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:06:10.123339833 +0000 UTC m=+525.052879484" watchObservedRunningTime="2026-07-17 11:06:10.124553411 +0000 UTC m=+525.054093074"
Jul 17 11:06:11.022262 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:11.022226    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-9w45x"]
Jul 17 11:06:12.114255 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:12.114221    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-9w45x" podUID="2dbbe88d-b695-481e-91b7-8853c333c8b3" containerName="authorino" containerID="cri-o://36aba5d1fce23b3f916bfc252617c0c7a5023ddf4c7dad47ebecf07569e0bd33" gracePeriod=30
Jul 17 11:06:12.365597 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:12.365528    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9w45x"
Jul 17 11:06:12.406384 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:12.406356    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5mhg\" (UniqueName: \"kubernetes.io/projected/2dbbe88d-b695-481e-91b7-8853c333c8b3-kube-api-access-g5mhg\") pod \"2dbbe88d-b695-481e-91b7-8853c333c8b3\" (UID: \"2dbbe88d-b695-481e-91b7-8853c333c8b3\") "
Jul 17 11:06:12.408495 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:12.408465    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dbbe88d-b695-481e-91b7-8853c333c8b3-kube-api-access-g5mhg" (OuterVolumeSpecName: "kube-api-access-g5mhg") pod "2dbbe88d-b695-481e-91b7-8853c333c8b3" (UID: "2dbbe88d-b695-481e-91b7-8853c333c8b3"). InnerVolumeSpecName "kube-api-access-g5mhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 11:06:12.507208 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:12.507171    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5mhg\" (UniqueName: \"kubernetes.io/projected/2dbbe88d-b695-481e-91b7-8853c333c8b3-kube-api-access-g5mhg\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:06:13.119829 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:13.119790    2563 generic.go:356] "Generic (PLEG): container finished" podID="2dbbe88d-b695-481e-91b7-8853c333c8b3" containerID="36aba5d1fce23b3f916bfc252617c0c7a5023ddf4c7dad47ebecf07569e0bd33" exitCode=0
Jul 17 11:06:13.120257 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:13.119838    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9w45x"
Jul 17 11:06:13.120257 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:13.119876    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9w45x" event={"ID":"2dbbe88d-b695-481e-91b7-8853c333c8b3","Type":"ContainerDied","Data":"36aba5d1fce23b3f916bfc252617c0c7a5023ddf4c7dad47ebecf07569e0bd33"}
Jul 17 11:06:13.120257 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:13.119923    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9w45x" event={"ID":"2dbbe88d-b695-481e-91b7-8853c333c8b3","Type":"ContainerDied","Data":"780e60430a8e014924caa66a72bad97a4f56c0854dc075e66d187d9990b15c13"}
Jul 17 11:06:13.120257 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:13.119940    2563 scope.go:117] "RemoveContainer" containerID="36aba5d1fce23b3f916bfc252617c0c7a5023ddf4c7dad47ebecf07569e0bd33"
Jul 17 11:06:13.129691 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:13.129669    2563 scope.go:117] "RemoveContainer" containerID="36aba5d1fce23b3f916bfc252617c0c7a5023ddf4c7dad47ebecf07569e0bd33"
Jul 17 11:06:13.129991 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:06:13.129968    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36aba5d1fce23b3f916bfc252617c0c7a5023ddf4c7dad47ebecf07569e0bd33\": container with ID starting with 36aba5d1fce23b3f916bfc252617c0c7a5023ddf4c7dad47ebecf07569e0bd33 not found: ID does not exist" containerID="36aba5d1fce23b3f916bfc252617c0c7a5023ddf4c7dad47ebecf07569e0bd33"
Jul 17 11:06:13.130089 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:13.130005    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36aba5d1fce23b3f916bfc252617c0c7a5023ddf4c7dad47ebecf07569e0bd33"} err="failed to get container status \"36aba5d1fce23b3f916bfc252617c0c7a5023ddf4c7dad47ebecf07569e0bd33\": rpc error: code = NotFound desc = could not find container \"36aba5d1fce23b3f916bfc252617c0c7a5023ddf4c7dad47ebecf07569e0bd33\": container with ID starting with 36aba5d1fce23b3f916bfc252617c0c7a5023ddf4c7dad47ebecf07569e0bd33 not found: ID does not exist"
Jul 17 11:06:13.148198 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:13.148156    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-9w45x"]
Jul 17 11:06:13.150274 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:13.150254    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-9w45x"]
Jul 17 11:06:13.664691 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:13.664659    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dbbe88d-b695-481e-91b7-8853c333c8b3" path="/var/lib/kubelet/pods/2dbbe88d-b695-481e-91b7-8853c333c8b3/volumes"
Jul 17 11:06:22.498865 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:22.498789    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:06:22.618536 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:22.618503    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:06:23.054451 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:23.054419    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:06:35.431260 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:35.431220    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:06:35.665702 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:35.665672    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ckd6r"]
Jul 17 11:06:35.665968 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:35.665956    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dbbe88d-b695-481e-91b7-8853c333c8b3" containerName="authorino"
Jul 17 11:06:35.666019 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:35.665970    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dbbe88d-b695-481e-91b7-8853c333c8b3" containerName="authorino"
Jul 17 11:06:35.666100 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:35.666092    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dbbe88d-b695-481e-91b7-8853c333c8b3" containerName="authorino"
Jul 17 11:06:35.670890 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:35.670872    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ckd6r"
Jul 17 11:06:35.677676 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:35.677655    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 17 11:06:35.678778 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:35.678757    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qm856\""
Jul 17 11:06:35.679171 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:35.679153    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 17 11:06:35.689364 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:35.689307    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ckd6r"]
Jul 17 11:06:35.722106 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:35.722062    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:06:35.806428 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:35.806385    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xml6v\" (UniqueName: \"kubernetes.io/projected/8a12060b-b56f-4c7e-bf5b-95adb120b9a0-kube-api-access-xml6v\") pod \"authorino-5d4b67c446-ckd6r\" (UID: \"8a12060b-b56f-4c7e-bf5b-95adb120b9a0\") " pod="kuadrant-system/authorino-5d4b67c446-ckd6r"
Jul 17 11:06:35.906999 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:35.906960    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xml6v\" (UniqueName: \"kubernetes.io/projected/8a12060b-b56f-4c7e-bf5b-95adb120b9a0-kube-api-access-xml6v\") pod \"authorino-5d4b67c446-ckd6r\" (UID: \"8a12060b-b56f-4c7e-bf5b-95adb120b9a0\") " pod="kuadrant-system/authorino-5d4b67c446-ckd6r"
Jul 17 11:06:35.913477 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:35.913446    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ckd6r"]
Jul 17 11:06:35.913688 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:06:35.913670    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-xml6v], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-ckd6r" podUID="8a12060b-b56f-4c7e-bf5b-95adb120b9a0"
Jul 17 11:06:35.919733 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:35.919710    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xml6v\" (UniqueName: \"kubernetes.io/projected/8a12060b-b56f-4c7e-bf5b-95adb120b9a0-kube-api-access-xml6v\") pod \"authorino-5d4b67c446-ckd6r\" (UID: \"8a12060b-b56f-4c7e-bf5b-95adb120b9a0\") " pod="kuadrant-system/authorino-5d4b67c446-ckd6r"
Jul 17 11:06:35.936216 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:35.936189    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-646847d9f6-42sf8"]
Jul 17 11:06:35.938769 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:35.938752    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-646847d9f6-42sf8"
Jul 17 11:06:35.951016 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:35.950990    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-646847d9f6-42sf8"]
Jul 17 11:06:36.109498 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:36.109461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v72c\" (UniqueName: \"kubernetes.io/projected/0429ee4b-de03-469e-a332-5c6b12c06924-kube-api-access-7v72c\") pod \"authorino-646847d9f6-42sf8\" (UID: \"0429ee4b-de03-469e-a332-5c6b12c06924\") " pod="kuadrant-system/authorino-646847d9f6-42sf8"
Jul 17 11:06:36.140196 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:36.140154    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-646847d9f6-42sf8"]
Jul 17 11:06:36.140724 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:06:36.140690    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-7v72c], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-646847d9f6-42sf8" podUID="0429ee4b-de03-469e-a332-5c6b12c06924"
Jul 17 11:06:36.167839 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:36.167801    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-568b6959b4-57sr6"]
Jul 17 11:06:36.170173 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:36.170155    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-568b6959b4-57sr6"
Jul 17 11:06:36.172926 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:36.172906    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 17 11:06:36.185232 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:36.185209    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-568b6959b4-57sr6"]
Jul 17 11:06:36.210981 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:36.210907    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v72c\" (UniqueName: \"kubernetes.io/projected/0429ee4b-de03-469e-a332-5c6b12c06924-kube-api-access-7v72c\") pod \"authorino-646847d9f6-42sf8\" (UID: \"0429ee4b-de03-469e-a332-5c6b12c06924\") " pod="kuadrant-system/authorino-646847d9f6-42sf8"
Jul 17 11:06:36.220480 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:36.220449    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v72c\" (UniqueName: \"kubernetes.io/projected/0429ee4b-de03-469e-a332-5c6b12c06924-kube-api-access-7v72c\") pod \"authorino-646847d9f6-42sf8\" (UID: \"0429ee4b-de03-469e-a332-5c6b12c06924\") " pod="kuadrant-system/authorino-646847d9f6-42sf8"
Jul 17 11:06:36.233053 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:36.233023    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-646847d9f6-42sf8"
Jul 17 11:06:36.233188 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:36.233028    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ckd6r"
Jul 17 11:06:36.239084 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:36.239054    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-646847d9f6-42sf8"
Jul 17 11:06:36.241781 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:36.241762    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ckd6r"
Jul 17 11:06:36.311781 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:36.311743    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eb96eabb-56df-46a8-a665-2f70e4a6a228-tls-cert\") pod \"authorino-568b6959b4-57sr6\" (UID: \"eb96eabb-56df-46a8-a665-2f70e4a6a228\") " pod="kuadrant-system/authorino-568b6959b4-57sr6"
Jul 17 11:06:36.311781 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:36.311784    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c84vx\" (UniqueName: \"kubernetes.io/projected/eb96eabb-56df-46a8-a665-2f70e4a6a228-kube-api-access-c84vx\") pod \"authorino-568b6959b4-57sr6\" (UID: \"eb96eabb-56df-46a8-a665-2f70e4a6a228\") " pod="kuadrant-system/authorino-568b6959b4-57sr6"
Jul 17 11:06:36.412264 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:36.412228    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v72c\" (UniqueName: \"kubernetes.io/projected/0429ee4b-de03-469e-a332-5c6b12c06924-kube-api-access-7v72c\") pod \"0429ee4b-de03-469e-a332-5c6b12c06924\" (UID: \"0429ee4b-de03-469e-a332-5c6b12c06924\") "
Jul 17 11:06:36.412409 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:36.412292    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xml6v\" (UniqueName: \"kubernetes.io/projected/8a12060b-b56f-4c7e-bf5b-95adb120b9a0-kube-api-access-xml6v\") pod \"8a12060b-b56f-4c7e-bf5b-95adb120b9a0\" (UID: \"8a12060b-b56f-4c7e-bf5b-95adb120b9a0\") "
Jul 17 11:06:36.412453 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:36.412433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eb96eabb-56df-46a8-a665-2f70e4a6a228-tls-cert\") pod \"authorino-568b6959b4-57sr6\" (UID: \"eb96eabb-56df-46a8-a665-2f70e4a6a228\") " pod="kuadrant-system/authorino-568b6959b4-57sr6"
Jul 17 11:06:36.412489 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:36.412457    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c84vx\" (UniqueName: \"kubernetes.io/projected/eb96eabb-56df-46a8-a665-2f70e4a6a228-kube-api-access-c84vx\") pod \"authorino-568b6959b4-57sr6\" (UID: \"eb96eabb-56df-46a8-a665-2f70e4a6a228\") " pod="kuadrant-system/authorino-568b6959b4-57sr6"
Jul 17 11:06:36.414471 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:36.414439    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0429ee4b-de03-469e-a332-5c6b12c06924-kube-api-access-7v72c" (OuterVolumeSpecName: "kube-api-access-7v72c") pod "0429ee4b-de03-469e-a332-5c6b12c06924" (UID: "0429ee4b-de03-469e-a332-5c6b12c06924"). InnerVolumeSpecName "kube-api-access-7v72c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 11:06:36.414650 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:36.414479    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a12060b-b56f-4c7e-bf5b-95adb120b9a0-kube-api-access-xml6v" (OuterVolumeSpecName: "kube-api-access-xml6v") pod "8a12060b-b56f-4c7e-bf5b-95adb120b9a0" (UID: "8a12060b-b56f-4c7e-bf5b-95adb120b9a0"). InnerVolumeSpecName "kube-api-access-xml6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 11:06:36.414716 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:36.414654    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eb96eabb-56df-46a8-a665-2f70e4a6a228-tls-cert\") pod \"authorino-568b6959b4-57sr6\" (UID: \"eb96eabb-56df-46a8-a665-2f70e4a6a228\") " pod="kuadrant-system/authorino-568b6959b4-57sr6"
Jul 17 11:06:36.422050 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:36.422011    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c84vx\" (UniqueName: \"kubernetes.io/projected/eb96eabb-56df-46a8-a665-2f70e4a6a228-kube-api-access-c84vx\") pod \"authorino-568b6959b4-57sr6\" (UID: \"eb96eabb-56df-46a8-a665-2f70e4a6a228\") " pod="kuadrant-system/authorino-568b6959b4-57sr6"
Jul 17 11:06:36.478458 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:36.478378    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-568b6959b4-57sr6"
Jul 17 11:06:36.513647 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:36.513619    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v72c\" (UniqueName: \"kubernetes.io/projected/0429ee4b-de03-469e-a332-5c6b12c06924-kube-api-access-7v72c\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:06:36.513647 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:36.513645    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xml6v\" (UniqueName: \"kubernetes.io/projected/8a12060b-b56f-4c7e-bf5b-95adb120b9a0-kube-api-access-xml6v\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:06:36.615877 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:36.615848    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-568b6959b4-57sr6"]
Jul 17 11:06:36.617745 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:06:36.617722    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb96eabb_56df_46a8_a665_2f70e4a6a228.slice/crio-41dba6618d3c398f7084a746fa96c2ab5a4d3542acff811dc18d0d58ec389973 WatchSource:0}: Error finding container 41dba6618d3c398f7084a746fa96c2ab5a4d3542acff811dc18d0d58ec389973: Status 404 returned error can't find the container with id 41dba6618d3c398f7084a746fa96c2ab5a4d3542acff811dc18d0d58ec389973
Jul 17 11:06:37.239358 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:37.239326    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ckd6r"
Jul 17 11:06:37.239358 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:37.239339    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-568b6959b4-57sr6" event={"ID":"eb96eabb-56df-46a8-a665-2f70e4a6a228","Type":"ContainerStarted","Data":"41dba6618d3c398f7084a746fa96c2ab5a4d3542acff811dc18d0d58ec389973"}
Jul 17 11:06:37.239358 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:37.239364    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-646847d9f6-42sf8"
Jul 17 11:06:37.283254 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:37.283222    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-646847d9f6-42sf8"]
Jul 17 11:06:37.289642 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:37.289613    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-646847d9f6-42sf8"]
Jul 17 11:06:37.320010 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:37.319979    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ckd6r"]
Jul 17 11:06:37.323315 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:37.323285    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ckd6r"]
Jul 17 11:06:37.665422 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:37.665384    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0429ee4b-de03-469e-a332-5c6b12c06924" path="/var/lib/kubelet/pods/0429ee4b-de03-469e-a332-5c6b12c06924/volumes"
Jul 17 11:06:37.665902 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:37.665700    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a12060b-b56f-4c7e-bf5b-95adb120b9a0" path="/var/lib/kubelet/pods/8a12060b-b56f-4c7e-bf5b-95adb120b9a0/volumes"
Jul 17 11:06:38.245779 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:38.245687    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-568b6959b4-57sr6" event={"ID":"eb96eabb-56df-46a8-a665-2f70e4a6a228","Type":"ContainerStarted","Data":"3a56d348b6bf031b14f02280668660d62e9e645d65ee6245156ee6df6a9c3fa9"}
Jul 17 11:06:38.265983 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:38.265932    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-568b6959b4-57sr6" podStartSLOduration=0.950544154 podStartE2EDuration="2.265917313s" totalImagesPullingTime="1.315373159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:06:38.26466077 +0000 UTC m=+553.194200413" watchObservedRunningTime="2026-07-17 11:06:38.265917313 +0000 UTC m=+553.195456953"
Jul 17 11:06:43.504325 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:43.504291    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:06:43.725790 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:43.725757    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:06:43.851568 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:43.851485    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:06:44.142567 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:44.142535    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:06:47.234763 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:47.234725    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:06:47.680708 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:47.680679    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:06:54.899392 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:06:54.899358    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:07:17.109901 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:07:17.109863    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:07:19.720357 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:07:19.720313    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:07:19.838261 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:07:19.838192    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:07:20.380214 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:07:20.380127    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:07:20.538008 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:07:20.537971    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:07:21.722151 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:07:21.722112    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:07:21.871685 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:07:21.871654    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:07:25.493214 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:07:25.493186    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nj5mj_55d1de37-8635-4de8-ab59-4ebfd0346a40/ovn-acl-logging/0.log"
Jul 17 11:07:25.497234 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:07:25.497213    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nj5mj_55d1de37-8635-4de8-ab59-4ebfd0346a40/ovn-acl-logging/0.log"
Jul 17 11:07:30.590657 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:07:30.590607    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:07:30.766518 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:07:30.766470    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:07:41.541310 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:07:41.541270    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:07:50.227905 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:07:50.227818    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:08:03.061017 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:03.060960    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:08:11.186561 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.186521    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs"]
Jul 17 11:08:11.189060 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.189039    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs"
Jul 17 11:08:11.193046 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.193027    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-db62k\""
Jul 17 11:08:11.193143 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.193095    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 17 11:08:11.193190 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.193138    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 17 11:08:11.193266 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.193254    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 17 11:08:11.200648 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.200628    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs"]
Jul 17 11:08:11.225715 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.225684    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:08:11.327602 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.327569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28fbea3e-f044-4134-a38d-bf84c02cb151-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs\" (UID: \"28fbea3e-f044-4134-a38d-bf84c02cb151\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs"
Jul 17 11:08:11.327789 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.327619    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28fbea3e-f044-4134-a38d-bf84c02cb151-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs\" (UID: \"28fbea3e-f044-4134-a38d-bf84c02cb151\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs"
Jul 17 11:08:11.327789 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.327684    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28fbea3e-f044-4134-a38d-bf84c02cb151-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs\" (UID: \"28fbea3e-f044-4134-a38d-bf84c02cb151\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs"
Jul 17 11:08:11.327789 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.327727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28fbea3e-f044-4134-a38d-bf84c02cb151-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs\" (UID: \"28fbea3e-f044-4134-a38d-bf84c02cb151\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs"
Jul 17 11:08:11.327789 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.327766    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzqb2\" (UniqueName: \"kubernetes.io/projected/28fbea3e-f044-4134-a38d-bf84c02cb151-kube-api-access-jzqb2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs\" (UID: \"28fbea3e-f044-4134-a38d-bf84c02cb151\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs"
Jul 17 11:08:11.327935 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.327831    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28fbea3e-f044-4134-a38d-bf84c02cb151-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs\" (UID: \"28fbea3e-f044-4134-a38d-bf84c02cb151\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs"
Jul 17 11:08:11.429188 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.429144    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28fbea3e-f044-4134-a38d-bf84c02cb151-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs\" (UID: \"28fbea3e-f044-4134-a38d-bf84c02cb151\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs"
Jul 17 11:08:11.429188 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.429191    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzqb2\" (UniqueName: \"kubernetes.io/projected/28fbea3e-f044-4134-a38d-bf84c02cb151-kube-api-access-jzqb2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs\" (UID: \"28fbea3e-f044-4134-a38d-bf84c02cb151\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs"
Jul 17 11:08:11.429444 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.429219    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28fbea3e-f044-4134-a38d-bf84c02cb151-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs\" (UID: \"28fbea3e-f044-4134-a38d-bf84c02cb151\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs"
Jul 17 11:08:11.429444 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.429262    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28fbea3e-f044-4134-a38d-bf84c02cb151-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs\" (UID: \"28fbea3e-f044-4134-a38d-bf84c02cb151\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs"
Jul 17 11:08:11.429444 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.429301    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28fbea3e-f044-4134-a38d-bf84c02cb151-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs\" (UID: \"28fbea3e-f044-4134-a38d-bf84c02cb151\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs"
Jul 17 11:08:11.429444 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.429330    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28fbea3e-f044-4134-a38d-bf84c02cb151-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs\" (UID: \"28fbea3e-f044-4134-a38d-bf84c02cb151\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs"
Jul 17 11:08:11.429658 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.429582    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28fbea3e-f044-4134-a38d-bf84c02cb151-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs\" (UID: \"28fbea3e-f044-4134-a38d-bf84c02cb151\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs"
Jul 17 11:08:11.429658 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.429625    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28fbea3e-f044-4134-a38d-bf84c02cb151-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs\" (UID: \"28fbea3e-f044-4134-a38d-bf84c02cb151\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs"
Jul 17 11:08:11.429743 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.429674    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28fbea3e-f044-4134-a38d-bf84c02cb151-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs\" (UID: \"28fbea3e-f044-4134-a38d-bf84c02cb151\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs"
Jul 17 11:08:11.431600 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.431578    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28fbea3e-f044-4134-a38d-bf84c02cb151-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs\" (UID: \"28fbea3e-f044-4134-a38d-bf84c02cb151\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs"
Jul 17 11:08:11.431763 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.431745    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28fbea3e-f044-4134-a38d-bf84c02cb151-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs\" (UID: \"28fbea3e-f044-4134-a38d-bf84c02cb151\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs"
Jul 17 11:08:11.440004 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.439945    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzqb2\" (UniqueName: \"kubernetes.io/projected/28fbea3e-f044-4134-a38d-bf84c02cb151-kube-api-access-jzqb2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs\" (UID: \"28fbea3e-f044-4134-a38d-bf84c02cb151\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs"
Jul 17 11:08:11.499866 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.499830    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs"
Jul 17 11:08:11.631180 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.631156    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs"]
Jul 17 11:08:11.632475 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:08:11.632453    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28fbea3e_f044_4134_a38d_bf84c02cb151.slice/crio-fcd00476c40ed2f87f69c417a881ae5006e4c5484ed338cc17800fa2f4801ef9 WatchSource:0}: Error finding container fcd00476c40ed2f87f69c417a881ae5006e4c5484ed338cc17800fa2f4801ef9: Status 404 returned error can't find the container with id fcd00476c40ed2f87f69c417a881ae5006e4c5484ed338cc17800fa2f4801ef9
Jul 17 11:08:11.633727 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.633709    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 11:08:11.694600 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:11.694518    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs" event={"ID":"28fbea3e-f044-4134-a38d-bf84c02cb151","Type":"ContainerStarted","Data":"fcd00476c40ed2f87f69c417a881ae5006e4c5484ed338cc17800fa2f4801ef9"}
Jul 17 11:08:12.702227 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:12.702196    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs" event={"ID":"28fbea3e-f044-4134-a38d-bf84c02cb151","Type":"ContainerStarted","Data":"2477a9a27347984d557639f486542cdf035eda2bcc8ed1cf4e4750201c97f9bf"}
Jul 17 11:08:12.725176 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:12.725109    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs" podStartSLOduration=0.757375279 podStartE2EDuration="1.725092099s" totalImagesPullingTime="967.71682ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:08:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:08:12.723704051 +0000 UTC m=+647.653243692" watchObservedRunningTime="2026-07-17 11:08:12.725092099 +0000 UTC m=+647.654631736"
Jul 17 11:08:15.293321 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:15.293277    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv"]
Jul 17 11:08:15.295853 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:15.295836    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv"
Jul 17 11:08:15.299205 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:15.299183    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 17 11:08:15.310161 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:15.310140    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv"]
Jul 17 11:08:15.328289 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:15.328264    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:08:15.466836 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:15.466798    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6995c1c-928c-415e-9353-99d9ed856ecf-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7jtv\" (UID: \"e6995c1c-928c-415e-9353-99d9ed856ecf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv"
Jul 17 11:08:15.467019 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:15.466851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jv4x\" (UniqueName: \"kubernetes.io/projected/e6995c1c-928c-415e-9353-99d9ed856ecf-kube-api-access-8jv4x\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7jtv\" (UID: \"e6995c1c-928c-415e-9353-99d9ed856ecf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv"
Jul 17 11:08:15.467019 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:15.466881    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6995c1c-928c-415e-9353-99d9ed856ecf-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7jtv\" (UID: \"e6995c1c-928c-415e-9353-99d9ed856ecf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv"
Jul 17 11:08:15.467019 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:15.466897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6995c1c-928c-415e-9353-99d9ed856ecf-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7jtv\" (UID: \"e6995c1c-928c-415e-9353-99d9ed856ecf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv"
Jul 17 11:08:15.467019 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:15.466916    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6995c1c-928c-415e-9353-99d9ed856ecf-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7jtv\" (UID: \"e6995c1c-928c-415e-9353-99d9ed856ecf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv"
Jul 17 11:08:15.467234 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:15.467009    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6995c1c-928c-415e-9353-99d9ed856ecf-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7jtv\" (UID: \"e6995c1c-928c-415e-9353-99d9ed856ecf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv"
Jul 17 11:08:15.567691 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:15.567612    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jv4x\" (UniqueName: \"kubernetes.io/projected/e6995c1c-928c-415e-9353-99d9ed856ecf-kube-api-access-8jv4x\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7jtv\" (UID: \"e6995c1c-928c-415e-9353-99d9ed856ecf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv"
Jul 17 11:08:15.567691 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:15.567648    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6995c1c-928c-415e-9353-99d9ed856ecf-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7jtv\" (UID: \"e6995c1c-928c-415e-9353-99d9ed856ecf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv"
Jul 17 11:08:15.567691 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:15.567666    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6995c1c-928c-415e-9353-99d9ed856ecf-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7jtv\" (UID: \"e6995c1c-928c-415e-9353-99d9ed856ecf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv"
Jul 17 11:08:15.567691 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:15.567683    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6995c1c-928c-415e-9353-99d9ed856ecf-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7jtv\" (UID: \"e6995c1c-928c-415e-9353-99d9ed856ecf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv"
Jul 17 11:08:15.567952 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:15.567744    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6995c1c-928c-415e-9353-99d9ed856ecf-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7jtv\" (UID: \"e6995c1c-928c-415e-9353-99d9ed856ecf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv"
Jul 17 11:08:15.567952 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:15.567799    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6995c1c-928c-415e-9353-99d9ed856ecf-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7jtv\" (UID: \"e6995c1c-928c-415e-9353-99d9ed856ecf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv"
Jul 17 11:08:15.568130 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:15.568113    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6995c1c-928c-415e-9353-99d9ed856ecf-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7jtv\" (UID: \"e6995c1c-928c-415e-9353-99d9ed856ecf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv"
Jul 17 11:08:15.568179 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:15.568157    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6995c1c-928c-415e-9353-99d9ed856ecf-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7jtv\" (UID: \"e6995c1c-928c-415e-9353-99d9ed856ecf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv"
Jul 17 11:08:15.568222 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:15.568206    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6995c1c-928c-415e-9353-99d9ed856ecf-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7jtv\" (UID: \"e6995c1c-928c-415e-9353-99d9ed856ecf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv"
Jul 17 11:08:15.569920 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:15.569899    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6995c1c-928c-415e-9353-99d9ed856ecf-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7jtv\" (UID: \"e6995c1c-928c-415e-9353-99d9ed856ecf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv"
Jul 17 11:08:15.570122 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:15.570103    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6995c1c-928c-415e-9353-99d9ed856ecf-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7jtv\" (UID: \"e6995c1c-928c-415e-9353-99d9ed856ecf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv"
Jul 17 11:08:15.577511 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:15.577491    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jv4x\" (UniqueName: \"kubernetes.io/projected/e6995c1c-928c-415e-9353-99d9ed856ecf-kube-api-access-8jv4x\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7jtv\" (UID: \"e6995c1c-928c-415e-9353-99d9ed856ecf\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv"
Jul 17 11:08:15.605386 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:15.605364    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv"
Jul 17 11:08:15.739676 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:15.739652    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv"]
Jul 17 11:08:15.740771 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:08:15.740745    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6995c1c_928c_415e_9353_99d9ed856ecf.slice/crio-e5846b384d0acd0f0de21fa667fd1a4d95312e3a9a18a53b5d99685a12b9dac7 WatchSource:0}: Error finding container e5846b384d0acd0f0de21fa667fd1a4d95312e3a9a18a53b5d99685a12b9dac7: Status 404 returned error can't find the container with id e5846b384d0acd0f0de21fa667fd1a4d95312e3a9a18a53b5d99685a12b9dac7
Jul 17 11:08:16.723630 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:16.723596    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv" event={"ID":"e6995c1c-928c-415e-9353-99d9ed856ecf","Type":"ContainerStarted","Data":"6dbb4f864e09e21564a9bde3b5562fb57a66dfb6460a7e6edb5c671a10065348"}
Jul 17 11:08:16.724036 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:16.723651    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv" event={"ID":"e6995c1c-928c-415e-9353-99d9ed856ecf","Type":"ContainerStarted","Data":"e5846b384d0acd0f0de21fa667fd1a4d95312e3a9a18a53b5d99685a12b9dac7"}
Jul 17 11:08:16.746052 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:16.745978    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv" podStartSLOduration=1.5692272 podStartE2EDuration="1.745963013s" totalImagesPullingTime="176.735813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:08:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:08:16.745155209 +0000 UTC m=+651.674694853" watchObservedRunningTime="2026-07-17 11:08:16.745963013 +0000 UTC m=+651.675502651"
Jul 17 11:08:18.227380 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:18.227345    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:08:21.500773 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:21.500729    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs"
Jul 17 11:08:21.501296 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:21.500800    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs"
Jul 17 11:08:21.513222 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:21.513197    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs"
Jul 17 11:08:21.763789 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:21.763695    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs"
Jul 17 11:08:25.606176 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:25.606138    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv"
Jul 17 11:08:25.606613 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:25.606210    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv"
Jul 17 11:08:25.618712 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:25.618687    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv"
Jul 17 11:08:25.781617 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:25.781589    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7jtv"
Jul 17 11:08:28.808634 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:28.808604    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7"]
Jul 17 11:08:28.811090 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:28.811058    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7"
Jul 17 11:08:28.813776 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:28.813749    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 17 11:08:28.826134 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:28.826110    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7"]
Jul 17 11:08:28.880523 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:28.880494    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51f661df-6c4f-47a8-8bff-244b58486a70-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7\" (UID: \"51f661df-6c4f-47a8-8bff-244b58486a70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7"
Jul 17 11:08:28.880698 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:28.880535    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51f661df-6c4f-47a8-8bff-244b58486a70-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7\" (UID: \"51f661df-6c4f-47a8-8bff-244b58486a70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7"
Jul 17 11:08:28.880698 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:28.880563    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f661df-6c4f-47a8-8bff-244b58486a70-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7\" (UID: \"51f661df-6c4f-47a8-8bff-244b58486a70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7"
Jul 17 11:08:28.880698 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:28.880603    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zxj9\" (UniqueName: \"kubernetes.io/projected/51f661df-6c4f-47a8-8bff-244b58486a70-kube-api-access-8zxj9\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7\" (UID: \"51f661df-6c4f-47a8-8bff-244b58486a70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7"
Jul 17 11:08:28.880698 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:28.880632    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51f661df-6c4f-47a8-8bff-244b58486a70-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7\" (UID: \"51f661df-6c4f-47a8-8bff-244b58486a70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7"
Jul 17 11:08:28.880698 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:28.880691    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51f661df-6c4f-47a8-8bff-244b58486a70-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7\" (UID: \"51f661df-6c4f-47a8-8bff-244b58486a70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7"
Jul 17 11:08:28.935634 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:28.935599    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:08:28.981713 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:28.981679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51f661df-6c4f-47a8-8bff-244b58486a70-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7\" (UID: \"51f661df-6c4f-47a8-8bff-244b58486a70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7"
Jul 17 11:08:28.981886 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:28.981727    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f661df-6c4f-47a8-8bff-244b58486a70-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7\" (UID: \"51f661df-6c4f-47a8-8bff-244b58486a70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7"
Jul 17 11:08:28.981886 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:28.981776    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zxj9\" (UniqueName: \"kubernetes.io/projected/51f661df-6c4f-47a8-8bff-244b58486a70-kube-api-access-8zxj9\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7\" (UID: \"51f661df-6c4f-47a8-8bff-244b58486a70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7"
Jul 17 11:08:28.981886 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:28.981821    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51f661df-6c4f-47a8-8bff-244b58486a70-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7\" (UID: \"51f661df-6c4f-47a8-8bff-244b58486a70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7"
Jul 17 11:08:28.981886 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:28.981855    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51f661df-6c4f-47a8-8bff-244b58486a70-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7\" (UID: \"51f661df-6c4f-47a8-8bff-244b58486a70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7"
Jul 17 11:08:28.982119 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:28.981905    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51f661df-6c4f-47a8-8bff-244b58486a70-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7\" (UID: \"51f661df-6c4f-47a8-8bff-244b58486a70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7"
Jul 17 11:08:28.982223 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:28.982199    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51f661df-6c4f-47a8-8bff-244b58486a70-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7\" (UID: \"51f661df-6c4f-47a8-8bff-244b58486a70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7"
Jul 17 11:08:28.982223 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:28.982215    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51f661df-6c4f-47a8-8bff-244b58486a70-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7\" (UID: \"51f661df-6c4f-47a8-8bff-244b58486a70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7"
Jul 17 11:08:28.982340 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:28.982320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51f661df-6c4f-47a8-8bff-244b58486a70-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7\" (UID: \"51f661df-6c4f-47a8-8bff-244b58486a70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7"
Jul 17 11:08:28.984140 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:28.984120    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51f661df-6c4f-47a8-8bff-244b58486a70-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7\" (UID: \"51f661df-6c4f-47a8-8bff-244b58486a70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7"
Jul 17 11:08:28.984276 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:28.984259    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f661df-6c4f-47a8-8bff-244b58486a70-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7\" (UID: \"51f661df-6c4f-47a8-8bff-244b58486a70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7"
Jul 17 11:08:28.994107 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:28.994085    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zxj9\" (UniqueName: \"kubernetes.io/projected/51f661df-6c4f-47a8-8bff-244b58486a70-kube-api-access-8zxj9\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7\" (UID: \"51f661df-6c4f-47a8-8bff-244b58486a70\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7"
Jul 17 11:08:29.120196 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:29.120107    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7"
Jul 17 11:08:29.256396 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:29.256372    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7"]
Jul 17 11:08:29.258241 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:08:29.258216    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51f661df_6c4f_47a8_8bff_244b58486a70.slice/crio-5ae2c2ca0f1f87b80c10ab0b491b567de4212820f720ac7f9c5b01725139c011 WatchSource:0}: Error finding container 5ae2c2ca0f1f87b80c10ab0b491b567de4212820f720ac7f9c5b01725139c011: Status 404 returned error can't find the container with id 5ae2c2ca0f1f87b80c10ab0b491b567de4212820f720ac7f9c5b01725139c011
Jul 17 11:08:29.792947 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:29.792844    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7" event={"ID":"51f661df-6c4f-47a8-8bff-244b58486a70","Type":"ContainerStarted","Data":"ebbe8ce4b52ccd917c290f34bf3cebc8de30ac88661fdabd13bf9b9ad4f6b2b7"}
Jul 17 11:08:29.792947 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:29.792899    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7" event={"ID":"51f661df-6c4f-47a8-8bff-244b58486a70","Type":"ContainerStarted","Data":"5ae2c2ca0f1f87b80c10ab0b491b567de4212820f720ac7f9c5b01725139c011"}
Jul 17 11:08:29.812571 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:29.812523    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7" podStartSLOduration=1.661345046 podStartE2EDuration="1.812508176s" totalImagesPullingTime="151.16313ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:08:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:08:29.810800245 +0000 UTC m=+664.740339897" watchObservedRunningTime="2026-07-17 11:08:29.812508176 +0000 UTC m=+664.742047814"
Jul 17 11:08:39.120878 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:39.120837    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7"
Jul 17 11:08:39.121688 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:39.121095    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7"
Jul 17 11:08:39.133482 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:39.133460    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7"
Jul 17 11:08:39.853298 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:39.853264    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7"
Jul 17 11:08:47.512564 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:47.512528    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:08:47.571836 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:47.571806    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:08:58.567056 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:08:58.567006    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:09:07.705569 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:07.705531    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c9d884449-brg67"]
Jul 17 11:09:07.711345 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:07.709565    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c9d884449-brg67"
Jul 17 11:09:07.716876 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:07.716850    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c9d884449-brg67"]
Jul 17 11:09:07.725949 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:07.725922    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02fad526-849d-4a1e-a418-af6f54952872-tls-cert\") pod \"authorino-c9d884449-brg67\" (UID: \"02fad526-849d-4a1e-a418-af6f54952872\") " pod="kuadrant-system/authorino-c9d884449-brg67"
Jul 17 11:09:07.726106 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:07.726007    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsl66\" (UniqueName: \"kubernetes.io/projected/02fad526-849d-4a1e-a418-af6f54952872-kube-api-access-gsl66\") pod \"authorino-c9d884449-brg67\" (UID: \"02fad526-849d-4a1e-a418-af6f54952872\") " pod="kuadrant-system/authorino-c9d884449-brg67"
Jul 17 11:09:07.826533 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:07.826493    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsl66\" (UniqueName: \"kubernetes.io/projected/02fad526-849d-4a1e-a418-af6f54952872-kube-api-access-gsl66\") pod \"authorino-c9d884449-brg67\" (UID: \"02fad526-849d-4a1e-a418-af6f54952872\") " pod="kuadrant-system/authorino-c9d884449-brg67"
Jul 17 11:09:07.826704 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:07.826572    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02fad526-849d-4a1e-a418-af6f54952872-tls-cert\") pod \"authorino-c9d884449-brg67\" (UID: \"02fad526-849d-4a1e-a418-af6f54952872\") " pod="kuadrant-system/authorino-c9d884449-brg67"
Jul 17 11:09:07.828949 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:07.828919    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02fad526-849d-4a1e-a418-af6f54952872-tls-cert\") pod \"authorino-c9d884449-brg67\" (UID: \"02fad526-849d-4a1e-a418-af6f54952872\") " pod="kuadrant-system/authorino-c9d884449-brg67"
Jul 17 11:09:07.836064 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:07.836033    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsl66\" (UniqueName: \"kubernetes.io/projected/02fad526-849d-4a1e-a418-af6f54952872-kube-api-access-gsl66\") pod \"authorino-c9d884449-brg67\" (UID: \"02fad526-849d-4a1e-a418-af6f54952872\") " pod="kuadrant-system/authorino-c9d884449-brg67"
Jul 17 11:09:08.022142 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:08.022048    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c9d884449-brg67"
Jul 17 11:09:08.176179 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:08.176143    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:09:08.211890 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:09:08.211855    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02fad526_849d_4a1e_a418_af6f54952872.slice/crio-85993c4610cace130b9369a0d14130705055065354d0c2068fcb0e088f08eeac WatchSource:0}: Error finding container 85993c4610cace130b9369a0d14130705055065354d0c2068fcb0e088f08eeac: Status 404 returned error can't find the container with id 85993c4610cace130b9369a0d14130705055065354d0c2068fcb0e088f08eeac
Jul 17 11:09:08.212817 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:08.212788    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c9d884449-brg67"]
Jul 17 11:09:08.364572 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:08.364438    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:09:08.989685 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:08.989566    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c9d884449-brg67" event={"ID":"02fad526-849d-4a1e-a418-af6f54952872","Type":"ContainerStarted","Data":"fb2abd2315bf9c4cd99729d5042d41afdaf98786a67aa48a37e30e03d26842ed"}
Jul 17 11:09:08.989685 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:08.989641    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c9d884449-brg67" event={"ID":"02fad526-849d-4a1e-a418-af6f54952872","Type":"ContainerStarted","Data":"85993c4610cace130b9369a0d14130705055065354d0c2068fcb0e088f08eeac"}
Jul 17 11:09:09.013614 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:09.013554    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c9d884449-brg67" podStartSLOduration=1.540925197 podStartE2EDuration="2.013533344s" totalImagesPullingTime="472.608147ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:09:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:09:09.01061895 +0000 UTC m=+703.940158592" watchObservedRunningTime="2026-07-17 11:09:09.013533344 +0000 UTC m=+703.943072986"
Jul 17 11:09:09.063789 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:09.061702    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-568b6959b4-57sr6"]
Jul 17 11:09:09.063789 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:09.061909    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-568b6959b4-57sr6" podUID="eb96eabb-56df-46a8-a665-2f70e4a6a228" containerName="authorino" containerID="cri-o://3a56d348b6bf031b14f02280668660d62e9e645d65ee6245156ee6df6a9c3fa9" gracePeriod=30
Jul 17 11:09:09.337127 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:09.337100    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-568b6959b4-57sr6"
Jul 17 11:09:09.343518 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:09.343481    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eb96eabb-56df-46a8-a665-2f70e4a6a228-tls-cert\") pod \"eb96eabb-56df-46a8-a665-2f70e4a6a228\" (UID: \"eb96eabb-56df-46a8-a665-2f70e4a6a228\") "
Jul 17 11:09:09.343667 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:09.343563    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c84vx\" (UniqueName: \"kubernetes.io/projected/eb96eabb-56df-46a8-a665-2f70e4a6a228-kube-api-access-c84vx\") pod \"eb96eabb-56df-46a8-a665-2f70e4a6a228\" (UID: \"eb96eabb-56df-46a8-a665-2f70e4a6a228\") "
Jul 17 11:09:09.345846 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:09.345815    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb96eabb-56df-46a8-a665-2f70e4a6a228-kube-api-access-c84vx" (OuterVolumeSpecName: "kube-api-access-c84vx") pod "eb96eabb-56df-46a8-a665-2f70e4a6a228" (UID: "eb96eabb-56df-46a8-a665-2f70e4a6a228"). InnerVolumeSpecName "kube-api-access-c84vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 11:09:09.354236 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:09.354211    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb96eabb-56df-46a8-a665-2f70e4a6a228-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "eb96eabb-56df-46a8-a665-2f70e4a6a228" (UID: "eb96eabb-56df-46a8-a665-2f70e4a6a228"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 11:09:09.445309 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:09.445272    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c84vx\" (UniqueName: \"kubernetes.io/projected/eb96eabb-56df-46a8-a665-2f70e4a6a228-kube-api-access-c84vx\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:09:09.445309 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:09.445302    2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eb96eabb-56df-46a8-a665-2f70e4a6a228-tls-cert\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:09:09.460515 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:09.460463    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:09:09.630847 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:09.630721    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:09:09.997693 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:09.997651    2563 generic.go:356] "Generic (PLEG): container finished" podID="eb96eabb-56df-46a8-a665-2f70e4a6a228" containerID="3a56d348b6bf031b14f02280668660d62e9e645d65ee6245156ee6df6a9c3fa9" exitCode=0
Jul 17 11:09:09.997693 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:09.997700    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-568b6959b4-57sr6"
Jul 17 11:09:09.998394 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:09.997748    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-568b6959b4-57sr6" event={"ID":"eb96eabb-56df-46a8-a665-2f70e4a6a228","Type":"ContainerDied","Data":"3a56d348b6bf031b14f02280668660d62e9e645d65ee6245156ee6df6a9c3fa9"}
Jul 17 11:09:09.998394 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:09.997797    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-568b6959b4-57sr6" event={"ID":"eb96eabb-56df-46a8-a665-2f70e4a6a228","Type":"ContainerDied","Data":"41dba6618d3c398f7084a746fa96c2ab5a4d3542acff811dc18d0d58ec389973"}
Jul 17 11:09:09.998394 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:09.997821    2563 scope.go:117] "RemoveContainer" containerID="3a56d348b6bf031b14f02280668660d62e9e645d65ee6245156ee6df6a9c3fa9"
Jul 17 11:09:10.006799 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:10.006773    2563 scope.go:117] "RemoveContainer" containerID="3a56d348b6bf031b14f02280668660d62e9e645d65ee6245156ee6df6a9c3fa9"
Jul 17 11:09:10.007097 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:09:10.007057    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a56d348b6bf031b14f02280668660d62e9e645d65ee6245156ee6df6a9c3fa9\": container with ID starting with 3a56d348b6bf031b14f02280668660d62e9e645d65ee6245156ee6df6a9c3fa9 not found: ID does not exist" containerID="3a56d348b6bf031b14f02280668660d62e9e645d65ee6245156ee6df6a9c3fa9"
Jul 17 11:09:10.007216 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:10.007103    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a56d348b6bf031b14f02280668660d62e9e645d65ee6245156ee6df6a9c3fa9"} err="failed to get container status \"3a56d348b6bf031b14f02280668660d62e9e645d65ee6245156ee6df6a9c3fa9\": rpc error: code = NotFound desc = could not find container \"3a56d348b6bf031b14f02280668660d62e9e645d65ee6245156ee6df6a9c3fa9\": container with ID starting with 3a56d348b6bf031b14f02280668660d62e9e645d65ee6245156ee6df6a9c3fa9 not found: ID does not exist"
Jul 17 11:09:10.038106 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:10.038051    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-568b6959b4-57sr6"]
Jul 17 11:09:10.046082 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:10.046034    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-568b6959b4-57sr6"]
Jul 17 11:09:11.665627 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:09:11.665591    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb96eabb-56df-46a8-a665-2f70e4a6a228" path="/var/lib/kubelet/pods/eb96eabb-56df-46a8-a665-2f70e4a6a228/volumes"
Jul 17 11:10:18.672422 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:10:18.672387    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:10:29.967127 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:10:29.967063    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:10:30.060761 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:10:30.060725    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:11:02.485025 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:11:02.484949    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:11:02.585750 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:11:02.585714    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:11:24.020740 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:11:24.020708    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:11:25.266777 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:11:25.266735    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:12:07.681489 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:12:07.681455    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:12:07.964429 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:12:07.964343    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:12:08.070293 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:12:08.070264    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:12:14.244742 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:12:14.244704    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:12:14.349866 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:12:14.349833    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:12:14.750935 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:12:14.750904    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:12:14.823018 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:12:14.822985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:12:20.863730 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:12:20.863651    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:12:20.947004 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:12:20.946936    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:12:25.557457 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:12:25.557419    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nj5mj_55d1de37-8635-4de8-ab59-4ebfd0346a40/ovn-acl-logging/0.log"
Jul 17 11:12:25.568421 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:12:25.567633    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nj5mj_55d1de37-8635-4de8-ab59-4ebfd0346a40/ovn-acl-logging/0.log"
Jul 17 11:14:26.824236 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:14:26.824162    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:17:16.108421 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:17:16.108390    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:17:20.488394 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:17:20.488362    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:17:25.601595 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:17:25.601566    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nj5mj_55d1de37-8635-4de8-ab59-4ebfd0346a40/ovn-acl-logging/0.log"
Jul 17 11:17:25.602153 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:17:25.602135    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nj5mj_55d1de37-8635-4de8-ab59-4ebfd0346a40/ovn-acl-logging/0.log"
Jul 17 11:20:46.683403 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:20:46.683368    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:20:48.344085 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:20:48.344042    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:22:25.633718 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:22:25.633688    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nj5mj_55d1de37-8635-4de8-ab59-4ebfd0346a40/ovn-acl-logging/0.log"
Jul 17 11:22:25.635882 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:22:25.635855    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nj5mj_55d1de37-8635-4de8-ab59-4ebfd0346a40/ovn-acl-logging/0.log"
Jul 17 11:23:15.336955 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:23:15.336916    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:23:15.421872 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:23:15.421841    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:23:20.962753 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:23:20.962720    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:23:21.063372 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:23:21.063339    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:23:21.178524 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:23:21.178466    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:23:21.280797 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:23:21.280696    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:23:22.403381 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:23:22.403335    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:23:22.500572 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:23:22.500527    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:23:22.707444 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:23:22.707357    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:23:22.872770 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:23:22.872738    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:23:23.268428 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:23:23.268391    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:23:23.366273 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:23:23.366226    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:23:25.214632 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:23:25.214603    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:24:30.000432 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:24:30.000327    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:24:35.561704 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:24:35.561674    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:25:40.926488 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:25:40.926449    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:26:27.667832 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:26:27.667796    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:26:29.047446 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:26:29.047409    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:26:34.511092 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:26:34.511033    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:26:35.234117 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:26:35.234061    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:26:41.320570 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:26:41.320536    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:26:41.404160 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:26:41.404112    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:26:41.630885 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:26:41.630796    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:26:41.724385 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:26:41.724354    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:26:41.844436 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:26:41.844401    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:26:43.203212 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:26:43.203177    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:26:43.326138 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:26:43.326106    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:26:43.651260 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:26:43.651227    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:26:44.045695 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:26:44.045611    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:26:44.153013 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:26:44.152978    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:26:49.089801 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:26:49.089767    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:26:49.190388 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:26:49.190356    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:26:49.339640 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:26:49.339612    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:26:49.435096 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:26:49.435051    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:26:50.074669 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:26:50.074633    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:26:50.173134 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:26:50.173098    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:26:52.578128 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:26:52.578089    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:27:25.665847 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:27:25.665820    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nj5mj_55d1de37-8635-4de8-ab59-4ebfd0346a40/ovn-acl-logging/0.log"
Jul 17 11:27:25.670109 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:27:25.670086    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nj5mj_55d1de37-8635-4de8-ab59-4ebfd0346a40/ovn-acl-logging/0.log"
Jul 17 11:27:47.869017 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:27:47.868979    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:27:47.989017 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:27:47.988988    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:27:54.047200 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:27:54.047165    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:27:54.299705 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:27:54.299617    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:27:54.398876 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:27:54.398837    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:28:00.171139 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:28:00.171086    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:28:00.271570 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:28:00.271536    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:28:00.403606 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:28:00.403574    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:28:00.517969 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:28:00.517882    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:28:01.527743 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:28:01.527651    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:28:01.617244 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:28:01.617203    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:28:01.753017 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:28:01.752948    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:28:02.035093 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:28:02.035040    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:28:02.440803 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:28:02.440763    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:28:02.537169 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:28:02.537135    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:28:03.871863 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:28:03.871829    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:28:57.843555 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:28:57.843471    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-88xqn"]
Jul 17 11:28:57.844008 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:28:57.843811    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb96eabb-56df-46a8-a665-2f70e4a6a228" containerName="authorino"
Jul 17 11:28:57.844008 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:28:57.843822    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb96eabb-56df-46a8-a665-2f70e4a6a228" containerName="authorino"
Jul 17 11:28:57.844008 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:28:57.843902    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb96eabb-56df-46a8-a665-2f70e4a6a228" containerName="authorino"
Jul 17 11:28:57.846813 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:28:57.846797    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-88xqn"
Jul 17 11:28:57.849532 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:28:57.849502    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 11:28:57.849650 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:28:57.849515    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwv2h\""
Jul 17 11:28:57.855978 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:28:57.855936    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-88xqn"]
Jul 17 11:28:57.998553 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:28:57.998514    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe43f0db-2cfe-4e75-9711-2de9eda96973-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-829f05b9-88xqn\" (UID: \"fe43f0db-2cfe-4e75-9711-2de9eda96973\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-88xqn"
Jul 17 11:28:58.099813 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:28:58.099728    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe43f0db-2cfe-4e75-9711-2de9eda96973-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-829f05b9-88xqn\" (UID: \"fe43f0db-2cfe-4e75-9711-2de9eda96973\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-88xqn"
Jul 17 11:28:58.100414 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:28:58.100393    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe43f0db-2cfe-4e75-9711-2de9eda96973-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-829f05b9-88xqn\" (UID: \"fe43f0db-2cfe-4e75-9711-2de9eda96973\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-88xqn"
Jul 17 11:28:58.158180 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:28:58.158149    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-88xqn"
Jul 17 11:28:58.497912 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:28:58.497885    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-88xqn"]
Jul 17 11:28:58.499661 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:28:58.499621    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe43f0db_2cfe_4e75_9711_2de9eda96973.slice/crio-30a6f50ecd707829e7530d337a56a54296a6990a5bd180db5bc7d7f3c8c86503 WatchSource:0}: Error finding container 30a6f50ecd707829e7530d337a56a54296a6990a5bd180db5bc7d7f3c8c86503: Status 404 returned error can't find the container with id 30a6f50ecd707829e7530d337a56a54296a6990a5bd180db5bc7d7f3c8c86503
Jul 17 11:28:58.501415 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:28:58.501401    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 11:28:58.901663 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:28:58.901626    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-88xqn" event={"ID":"fe43f0db-2cfe-4e75-9711-2de9eda96973","Type":"ContainerStarted","Data":"30a6f50ecd707829e7530d337a56a54296a6990a5bd180db5bc7d7f3c8c86503"}
Jul 17 11:29:00.913431 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:00.913352    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-88xqn" event={"ID":"fe43f0db-2cfe-4e75-9711-2de9eda96973","Type":"ContainerStarted","Data":"03f9182e74d872a79f9a18c4df0b68ded012eb8d163b374e587ff1f37523d28a"}
Jul 17 11:29:00.930845 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:00.930795    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-88xqn" podStartSLOduration=1.778523119 podStartE2EDuration="3.93078031s" totalImagesPullingTime="2.152257191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:28:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:29:00.929711286 +0000 UTC m=+1895.859250925" watchObservedRunningTime="2026-07-17 11:29:00.93078031 +0000 UTC m=+1895.860319952"
Jul 17 11:29:01.919482 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:01.919450    2563 generic.go:356] "Generic (PLEG): container finished" podID="fe43f0db-2cfe-4e75-9711-2de9eda96973" containerID="03f9182e74d872a79f9a18c4df0b68ded012eb8d163b374e587ff1f37523d28a" exitCode=6
Jul 17 11:29:01.919951 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:01.919535    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-88xqn" event={"ID":"fe43f0db-2cfe-4e75-9711-2de9eda96973","Type":"ContainerDied","Data":"03f9182e74d872a79f9a18c4df0b68ded012eb8d163b374e587ff1f37523d28a"}
Jul 17 11:29:01.919951 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:01.919710    2563 scope.go:117] "RemoveContainer" containerID="03f9182e74d872a79f9a18c4df0b68ded012eb8d163b374e587ff1f37523d28a"
Jul 17 11:29:02.926297 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:02.926260    2563 generic.go:356] "Generic (PLEG): container finished" podID="fe43f0db-2cfe-4e75-9711-2de9eda96973" containerID="b67fbbe27dfae29ce542a5e1af166f25ee20db1b76aa97425b4534f7bd08c01b" exitCode=6
Jul 17 11:29:02.926780 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:02.926346    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-88xqn" event={"ID":"fe43f0db-2cfe-4e75-9711-2de9eda96973","Type":"ContainerDied","Data":"b67fbbe27dfae29ce542a5e1af166f25ee20db1b76aa97425b4534f7bd08c01b"}
Jul 17 11:29:02.926780 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:02.926419    2563 scope.go:117] "RemoveContainer" containerID="03f9182e74d872a79f9a18c4df0b68ded012eb8d163b374e587ff1f37523d28a"
Jul 17 11:29:02.926780 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:02.926538    2563 scope.go:117] "RemoveContainer" containerID="b67fbbe27dfae29ce542a5e1af166f25ee20db1b76aa97425b4534f7bd08c01b"
Jul 17 11:29:02.926780 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:29:02.926744    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-829f05b9-88xqn_odh-ai-gateway-infra(fe43f0db-2cfe-4e75-9711-2de9eda96973)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-88xqn" podUID="fe43f0db-2cfe-4e75-9711-2de9eda96973"
Jul 17 11:29:03.933013 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:03.932990    2563 scope.go:117] "RemoveContainer" containerID="b67fbbe27dfae29ce542a5e1af166f25ee20db1b76aa97425b4534f7bd08c01b"
Jul 17 11:29:03.933387 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:29:03.933224    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-829f05b9-88xqn_odh-ai-gateway-infra(fe43f0db-2cfe-4e75-9711-2de9eda96973)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-88xqn" podUID="fe43f0db-2cfe-4e75-9711-2de9eda96973"
Jul 17 11:29:04.949824 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:04.949791    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:29:18.660443 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:18.660404    2563 scope.go:117] "RemoveContainer" containerID="b67fbbe27dfae29ce542a5e1af166f25ee20db1b76aa97425b4534f7bd08c01b"
Jul 17 11:29:19.011808 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:19.011723    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-88xqn" event={"ID":"fe43f0db-2cfe-4e75-9711-2de9eda96973","Type":"ContainerStarted","Data":"7381de6724ea62805feeb6e15e2d29fb2fd952bf734d4996d4a392aa2969bcf7"}
Jul 17 11:29:20.017886 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:20.017850    2563 generic.go:356] "Generic (PLEG): container finished" podID="fe43f0db-2cfe-4e75-9711-2de9eda96973" containerID="7381de6724ea62805feeb6e15e2d29fb2fd952bf734d4996d4a392aa2969bcf7" exitCode=6
Jul 17 11:29:20.018406 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:20.017925    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-88xqn" event={"ID":"fe43f0db-2cfe-4e75-9711-2de9eda96973","Type":"ContainerDied","Data":"7381de6724ea62805feeb6e15e2d29fb2fd952bf734d4996d4a392aa2969bcf7"}
Jul 17 11:29:20.018406 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:20.018009    2563 scope.go:117] "RemoveContainer" containerID="b67fbbe27dfae29ce542a5e1af166f25ee20db1b76aa97425b4534f7bd08c01b"
Jul 17 11:29:20.018406 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:20.018350    2563 scope.go:117] "RemoveContainer" containerID="7381de6724ea62805feeb6e15e2d29fb2fd952bf734d4996d4a392aa2969bcf7"
Jul 17 11:29:20.018562 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:29:20.018541    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-829f05b9-88xqn_odh-ai-gateway-infra(fe43f0db-2cfe-4e75-9711-2de9eda96973)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-88xqn" podUID="fe43f0db-2cfe-4e75-9711-2de9eda96973"
Jul 17 11:29:20.046904 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:20.046870    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-88xqn"]
Jul 17 11:29:21.147250 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:21.147227    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-88xqn"
Jul 17 11:29:21.211544 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:21.211516    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe43f0db-2cfe-4e75-9711-2de9eda96973-maas-api-service-ca\") pod \"fe43f0db-2cfe-4e75-9711-2de9eda96973\" (UID: \"fe43f0db-2cfe-4e75-9711-2de9eda96973\") "
Jul 17 11:29:21.211930 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:21.211907    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe43f0db-2cfe-4e75-9711-2de9eda96973-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fe43f0db-2cfe-4e75-9711-2de9eda96973" (UID: "fe43f0db-2cfe-4e75-9711-2de9eda96973"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:29:21.312884 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:21.312802    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe43f0db-2cfe-4e75-9711-2de9eda96973-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:29:22.029453 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:22.029375    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-88xqn"
Jul 17 11:29:22.029453 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:22.029395    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-88xqn" event={"ID":"fe43f0db-2cfe-4e75-9711-2de9eda96973","Type":"ContainerDied","Data":"30a6f50ecd707829e7530d337a56a54296a6990a5bd180db5bc7d7f3c8c86503"}
Jul 17 11:29:22.029646 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:22.029459    2563 scope.go:117] "RemoveContainer" containerID="7381de6724ea62805feeb6e15e2d29fb2fd952bf734d4996d4a392aa2969bcf7"
Jul 17 11:29:22.053425 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:22.053393    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-88xqn"]
Jul 17 11:29:22.058103 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:22.058059    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-88xqn"]
Jul 17 11:29:23.665348 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:23.665312    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe43f0db-2cfe-4e75-9711-2de9eda96973" path="/var/lib/kubelet/pods/fe43f0db-2cfe-4e75-9711-2de9eda96973/volumes"
Jul 17 11:29:37.848360 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:37.848328    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz"]
Jul 17 11:29:37.848811 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:37.848645    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe43f0db-2cfe-4e75-9711-2de9eda96973" containerName="revoke-keys"
Jul 17 11:29:37.848811 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:37.848656    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe43f0db-2cfe-4e75-9711-2de9eda96973" containerName="revoke-keys"
Jul 17 11:29:37.848811 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:37.848673    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe43f0db-2cfe-4e75-9711-2de9eda96973" containerName="revoke-keys"
Jul 17 11:29:37.848811 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:37.848678    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe43f0db-2cfe-4e75-9711-2de9eda96973" containerName="revoke-keys"
Jul 17 11:29:37.848811 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:37.848739    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe43f0db-2cfe-4e75-9711-2de9eda96973" containerName="revoke-keys"
Jul 17 11:29:37.848811 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:37.848745    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe43f0db-2cfe-4e75-9711-2de9eda96973" containerName="revoke-keys"
Jul 17 11:29:37.848811 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:37.848751    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe43f0db-2cfe-4e75-9711-2de9eda96973" containerName="revoke-keys"
Jul 17 11:29:37.858003 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:37.857983    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz"
Jul 17 11:29:37.861776 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:37.861749    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwv2h\""
Jul 17 11:29:37.861895 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:37.861801    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 11:29:37.862715 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:37.862691    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz"]
Jul 17 11:29:37.962296 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:37.962251    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33863eb2-7000-445d-bfaf-9230a409ef0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz\" (UID: \"33863eb2-7000-445d-bfaf-9230a409ef0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz"
Jul 17 11:29:38.063258 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:38.063224    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33863eb2-7000-445d-bfaf-9230a409ef0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz\" (UID: \"33863eb2-7000-445d-bfaf-9230a409ef0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz"
Jul 17 11:29:38.063864 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:38.063826    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33863eb2-7000-445d-bfaf-9230a409ef0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz\" (UID: \"33863eb2-7000-445d-bfaf-9230a409ef0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz"
Jul 17 11:29:38.172663 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:38.172631    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz"
Jul 17 11:29:38.315499 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:38.315439    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz"]
Jul 17 11:29:38.318184 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:29:38.318161    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33863eb2_7000_445d_bfaf_9230a409ef0e.slice/crio-c06ae899baa5de0d37de59a3dbf631fdd889b2bf8691cb0c155ac3078390f612 WatchSource:0}: Error finding container c06ae899baa5de0d37de59a3dbf631fdd889b2bf8691cb0c155ac3078390f612: Status 404 returned error can't find the container with id c06ae899baa5de0d37de59a3dbf631fdd889b2bf8691cb0c155ac3078390f612
Jul 17 11:29:39.115286 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:39.115250    2563 generic.go:356] "Generic (PLEG): container finished" podID="33863eb2-7000-445d-bfaf-9230a409ef0e" containerID="08d93f7e4f10aea8584a710eda4a519d170dbc47c6379eeb82f15b1ef3b647f1" exitCode=6
Jul 17 11:29:39.115709 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:39.115317    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz" event={"ID":"33863eb2-7000-445d-bfaf-9230a409ef0e","Type":"ContainerDied","Data":"08d93f7e4f10aea8584a710eda4a519d170dbc47c6379eeb82f15b1ef3b647f1"}
Jul 17 11:29:39.115709 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:39.115359    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz" event={"ID":"33863eb2-7000-445d-bfaf-9230a409ef0e","Type":"ContainerStarted","Data":"c06ae899baa5de0d37de59a3dbf631fdd889b2bf8691cb0c155ac3078390f612"}
Jul 17 11:29:39.115709 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:39.115531    2563 scope.go:117] "RemoveContainer" containerID="08d93f7e4f10aea8584a710eda4a519d170dbc47c6379eeb82f15b1ef3b647f1"
Jul 17 11:29:40.122582 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:40.122488    2563 generic.go:356] "Generic (PLEG): container finished" podID="33863eb2-7000-445d-bfaf-9230a409ef0e" containerID="c01551b4d4b35f2cf76fbdb99adebce7ef3f553f88074b4a99c6a909e0f1df87" exitCode=6
Jul 17 11:29:40.122582 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:40.122569    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz" event={"ID":"33863eb2-7000-445d-bfaf-9230a409ef0e","Type":"ContainerDied","Data":"c01551b4d4b35f2cf76fbdb99adebce7ef3f553f88074b4a99c6a909e0f1df87"}
Jul 17 11:29:40.123108 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:40.122629    2563 scope.go:117] "RemoveContainer" containerID="08d93f7e4f10aea8584a710eda4a519d170dbc47c6379eeb82f15b1ef3b647f1"
Jul 17 11:29:40.123108 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:40.122763    2563 scope.go:117] "RemoveContainer" containerID="c01551b4d4b35f2cf76fbdb99adebce7ef3f553f88074b4a99c6a909e0f1df87"
Jul 17 11:29:40.123108 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:29:40.122988    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz_odh-ai-gateway-infra(33863eb2-7000-445d-bfaf-9230a409ef0e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz" podUID="33863eb2-7000-445d-bfaf-9230a409ef0e"
Jul 17 11:29:41.130876 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:41.130846    2563 scope.go:117] "RemoveContainer" containerID="c01551b4d4b35f2cf76fbdb99adebce7ef3f553f88074b4a99c6a909e0f1df87"
Jul 17 11:29:41.131295 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:29:41.131095    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz_odh-ai-gateway-infra(33863eb2-7000-445d-bfaf-9230a409ef0e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz" podUID="33863eb2-7000-445d-bfaf-9230a409ef0e"
Jul 17 11:29:52.659900 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:52.659866    2563 scope.go:117] "RemoveContainer" containerID="c01551b4d4b35f2cf76fbdb99adebce7ef3f553f88074b4a99c6a909e0f1df87"
Jul 17 11:29:53.192242 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:53.192206    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz" event={"ID":"33863eb2-7000-445d-bfaf-9230a409ef0e","Type":"ContainerStarted","Data":"47f8d9f762c81d8921b98bf7a2099b0d561990a57811c03a2d17ea04613670ee"}
Jul 17 11:29:53.213383 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:53.213285    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz" podStartSLOduration=16.213270531 podStartE2EDuration="16.213270531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:29:53.211213945 +0000 UTC m=+1948.140753588" watchObservedRunningTime="2026-07-17 11:29:53.213270531 +0000 UTC m=+1948.142810171"
Jul 17 11:29:54.199104 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:54.199054    2563 generic.go:356] "Generic (PLEG): container finished" podID="33863eb2-7000-445d-bfaf-9230a409ef0e" containerID="47f8d9f762c81d8921b98bf7a2099b0d561990a57811c03a2d17ea04613670ee" exitCode=6
Jul 17 11:29:54.199569 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:54.199125    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz" event={"ID":"33863eb2-7000-445d-bfaf-9230a409ef0e","Type":"ContainerDied","Data":"47f8d9f762c81d8921b98bf7a2099b0d561990a57811c03a2d17ea04613670ee"}
Jul 17 11:29:54.199569 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:54.199197    2563 scope.go:117] "RemoveContainer" containerID="c01551b4d4b35f2cf76fbdb99adebce7ef3f553f88074b4a99c6a909e0f1df87"
Jul 17 11:29:54.199569 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:54.199422    2563 scope.go:117] "RemoveContainer" containerID="47f8d9f762c81d8921b98bf7a2099b0d561990a57811c03a2d17ea04613670ee"
Jul 17 11:29:54.199724 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:29:54.199624    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz_odh-ai-gateway-infra(33863eb2-7000-445d-bfaf-9230a409ef0e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz" podUID="33863eb2-7000-445d-bfaf-9230a409ef0e"
Jul 17 11:29:54.222936 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:54.222908    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz"]
Jul 17 11:29:55.333892 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:55.333866    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz"
Jul 17 11:29:55.518109 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:55.517974    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33863eb2-7000-445d-bfaf-9230a409ef0e-maas-api-service-ca\") pod \"33863eb2-7000-445d-bfaf-9230a409ef0e\" (UID: \"33863eb2-7000-445d-bfaf-9230a409ef0e\") "
Jul 17 11:29:55.518363 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:55.518339    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33863eb2-7000-445d-bfaf-9230a409ef0e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "33863eb2-7000-445d-bfaf-9230a409ef0e" (UID: "33863eb2-7000-445d-bfaf-9230a409ef0e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:29:55.618718 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:55.618689    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33863eb2-7000-445d-bfaf-9230a409ef0e-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:29:56.212320 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:56.212289    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz"
Jul 17 11:29:56.212496 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:56.212291    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz" event={"ID":"33863eb2-7000-445d-bfaf-9230a409ef0e","Type":"ContainerDied","Data":"c06ae899baa5de0d37de59a3dbf631fdd889b2bf8691cb0c155ac3078390f612"}
Jul 17 11:29:56.212496 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:56.212428    2563 scope.go:117] "RemoveContainer" containerID="47f8d9f762c81d8921b98bf7a2099b0d561990a57811c03a2d17ea04613670ee"
Jul 17 11:29:56.232160 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:56.232132    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz"]
Jul 17 11:29:56.238615 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:56.238579    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-gwqkz"]
Jul 17 11:29:57.665024 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:29:57.664991    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33863eb2-7000-445d-bfaf-9230a409ef0e" path="/var/lib/kubelet/pods/33863eb2-7000-445d-bfaf-9230a409ef0e/volumes"
Jul 17 11:30:00.147225 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:00.147172    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-45989be5-29738130-gtbt9"]
Jul 17 11:30:00.147727 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:00.147709    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33863eb2-7000-445d-bfaf-9230a409ef0e" containerName="revoke-keys"
Jul 17 11:30:00.147789 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:00.147731    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="33863eb2-7000-445d-bfaf-9230a409ef0e" containerName="revoke-keys"
Jul 17 11:30:00.147789 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:00.147743    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33863eb2-7000-445d-bfaf-9230a409ef0e" containerName="revoke-keys"
Jul 17 11:30:00.147789 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:00.147752    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="33863eb2-7000-445d-bfaf-9230a409ef0e" containerName="revoke-keys"
Jul 17 11:30:00.147789 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:00.147760    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33863eb2-7000-445d-bfaf-9230a409ef0e" containerName="revoke-keys"
Jul 17 11:30:00.147789 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:00.147766    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="33863eb2-7000-445d-bfaf-9230a409ef0e" containerName="revoke-keys"
Jul 17 11:30:00.147789 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:00.147774    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe43f0db-2cfe-4e75-9711-2de9eda96973" containerName="revoke-keys"
Jul 17 11:30:00.147789 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:00.147779    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe43f0db-2cfe-4e75-9711-2de9eda96973" containerName="revoke-keys"
Jul 17 11:30:00.147992 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:00.147834    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="33863eb2-7000-445d-bfaf-9230a409ef0e" containerName="revoke-keys"
Jul 17 11:30:00.147992 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:00.147846    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="33863eb2-7000-445d-bfaf-9230a409ef0e" containerName="revoke-keys"
Jul 17 11:30:00.151294 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:00.151275    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-45989be5-29738130-gtbt9"
Jul 17 11:30:00.154341 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:00.154315    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwv2h\""
Jul 17 11:30:00.161691 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:00.161666    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-45989be5-29738130-gtbt9"]
Jul 17 11:30:00.162232 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:00.162212    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-45989be5-29738130-gtbt9"
Jul 17 11:30:00.302682 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:00.302655    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-45989be5-29738130-gtbt9"]
Jul 17 11:30:00.304386 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:30:00.304360    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1f3d1a_3272_4d42_9e15_cfaad6f17ee1.slice/crio-93ead85e1419cd628d7a4c127e3e12df3ac490d67f80bdf21e3af903d6922ecb WatchSource:0}: Error finding container 93ead85e1419cd628d7a4c127e3e12df3ac490d67f80bdf21e3af903d6922ecb: Status 404 returned error can't find the container with id 93ead85e1419cd628d7a4c127e3e12df3ac490d67f80bdf21e3af903d6922ecb
Jul 17 11:30:01.240490 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:01.240457    2563 generic.go:356] "Generic (PLEG): container finished" podID="4b1f3d1a-3272-4d42-9e15-cfaad6f17ee1" containerID="8b0782e2c943b990b0110c12823b709ed77bcb6933df3a35a343fdc3d6a456e7" exitCode=7
Jul 17 11:30:01.240906 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:01.240516    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-45989be5-29738130-gtbt9" event={"ID":"4b1f3d1a-3272-4d42-9e15-cfaad6f17ee1","Type":"ContainerDied","Data":"8b0782e2c943b990b0110c12823b709ed77bcb6933df3a35a343fdc3d6a456e7"}
Jul 17 11:30:01.240906 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:01.240554    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-45989be5-29738130-gtbt9" event={"ID":"4b1f3d1a-3272-4d42-9e15-cfaad6f17ee1","Type":"ContainerStarted","Data":"93ead85e1419cd628d7a4c127e3e12df3ac490d67f80bdf21e3af903d6922ecb"}
Jul 17 11:30:01.240906 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:01.240655    2563 scope.go:117] "RemoveContainer" containerID="8b0782e2c943b990b0110c12823b709ed77bcb6933df3a35a343fdc3d6a456e7"
Jul 17 11:30:02.247295 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:02.247211    2563 generic.go:356] "Generic (PLEG): container finished" podID="4b1f3d1a-3272-4d42-9e15-cfaad6f17ee1" containerID="f1687d45a9e1713364f90c169664ae19bcc320eac5367b20cbe1b9a301ccb5eb" exitCode=7
Jul 17 11:30:02.247672 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:02.247292    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-45989be5-29738130-gtbt9" event={"ID":"4b1f3d1a-3272-4d42-9e15-cfaad6f17ee1","Type":"ContainerDied","Data":"f1687d45a9e1713364f90c169664ae19bcc320eac5367b20cbe1b9a301ccb5eb"}
Jul 17 11:30:02.247672 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:02.247351    2563 scope.go:117] "RemoveContainer" containerID="8b0782e2c943b990b0110c12823b709ed77bcb6933df3a35a343fdc3d6a456e7"
Jul 17 11:30:02.247672 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:02.247418    2563 scope.go:117] "RemoveContainer" containerID="f1687d45a9e1713364f90c169664ae19bcc320eac5367b20cbe1b9a301ccb5eb"
Jul 17 11:30:02.247786 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:30:02.247677    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-45989be5-29738130-gtbt9_odh-ai-gateway-infra(4b1f3d1a-3272-4d42-9e15-cfaad6f17ee1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-45989be5-29738130-gtbt9" podUID="4b1f3d1a-3272-4d42-9e15-cfaad6f17ee1"
Jul 17 11:30:03.254372 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:03.254346    2563 scope.go:117] "RemoveContainer" containerID="f1687d45a9e1713364f90c169664ae19bcc320eac5367b20cbe1b9a301ccb5eb"
Jul 17 11:30:03.254734 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:30:03.254556    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-45989be5-29738130-gtbt9_odh-ai-gateway-infra(4b1f3d1a-3272-4d42-9e15-cfaad6f17ee1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-45989be5-29738130-gtbt9" podUID="4b1f3d1a-3272-4d42-9e15-cfaad6f17ee1"
Jul 17 11:30:14.659801 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:14.659770    2563 scope.go:117] "RemoveContainer" containerID="f1687d45a9e1713364f90c169664ae19bcc320eac5367b20cbe1b9a301ccb5eb"
Jul 17 11:30:15.320893 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:15.320811    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-45989be5-29738130-gtbt9" event={"ID":"4b1f3d1a-3272-4d42-9e15-cfaad6f17ee1","Type":"ContainerStarted","Data":"462ad14fa5b88b9923a630ea3970b801de0d6ccc86c7afc0653b3d35aedd99d1"}
Jul 17 11:30:15.348788 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:15.348741    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-45989be5-29738130-gtbt9" podStartSLOduration=15.348720968 podStartE2EDuration="15.348720968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:30:15.347447998 +0000 UTC m=+1970.276987640" watchObservedRunningTime="2026-07-17 11:30:15.348720968 +0000 UTC m=+1970.278260612"
Jul 17 11:30:16.327198 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:16.327164    2563 generic.go:356] "Generic (PLEG): container finished" podID="4b1f3d1a-3272-4d42-9e15-cfaad6f17ee1" containerID="462ad14fa5b88b9923a630ea3970b801de0d6ccc86c7afc0653b3d35aedd99d1" exitCode=7
Jul 17 11:30:16.327575 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:16.327225    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-45989be5-29738130-gtbt9" event={"ID":"4b1f3d1a-3272-4d42-9e15-cfaad6f17ee1","Type":"ContainerDied","Data":"462ad14fa5b88b9923a630ea3970b801de0d6ccc86c7afc0653b3d35aedd99d1"}
Jul 17 11:30:16.327575 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:16.327303    2563 scope.go:117] "RemoveContainer" containerID="f1687d45a9e1713364f90c169664ae19bcc320eac5367b20cbe1b9a301ccb5eb"
Jul 17 11:30:16.327684 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:16.327584    2563 scope.go:117] "RemoveContainer" containerID="462ad14fa5b88b9923a630ea3970b801de0d6ccc86c7afc0653b3d35aedd99d1"
Jul 17 11:30:16.327874 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:30:16.327826    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-45989be5-29738130-gtbt9_odh-ai-gateway-infra(4b1f3d1a-3272-4d42-9e15-cfaad6f17ee1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-45989be5-29738130-gtbt9" podUID="4b1f3d1a-3272-4d42-9e15-cfaad6f17ee1"
Jul 17 11:30:16.366724 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:16.366692    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-45989be5-29738130-gtbt9"]
Jul 17 11:30:17.467744 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:17.467717    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-45989be5-29738130-gtbt9"
Jul 17 11:30:18.340230 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:18.340200    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-45989be5-29738130-gtbt9"
Jul 17 11:30:18.340448 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:18.340201    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-45989be5-29738130-gtbt9" event={"ID":"4b1f3d1a-3272-4d42-9e15-cfaad6f17ee1","Type":"ContainerDied","Data":"93ead85e1419cd628d7a4c127e3e12df3ac490d67f80bdf21e3af903d6922ecb"}
Jul 17 11:30:18.340448 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:18.340349    2563 scope.go:117] "RemoveContainer" containerID="462ad14fa5b88b9923a630ea3970b801de0d6ccc86c7afc0653b3d35aedd99d1"
Jul 17 11:30:18.361211 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:18.361169    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-45989be5-29738130-gtbt9"]
Jul 17 11:30:18.364162 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:18.364134    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-45989be5-29738130-gtbt9"]
Jul 17 11:30:19.665091 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:19.665048    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b1f3d1a-3272-4d42-9e15-cfaad6f17ee1" path="/var/lib/kubelet/pods/4b1f3d1a-3272-4d42-9e15-cfaad6f17ee1/volumes"
Jul 17 11:30:27.913020 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:27.912982    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-mb5qc"]
Jul 17 11:30:27.913549 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:27.913469    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b1f3d1a-3272-4d42-9e15-cfaad6f17ee1" containerName="cleanup"
Jul 17 11:30:27.913549 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:27.913487    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b1f3d1a-3272-4d42-9e15-cfaad6f17ee1" containerName="cleanup"
Jul 17 11:30:27.913549 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:27.913509    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b1f3d1a-3272-4d42-9e15-cfaad6f17ee1" containerName="cleanup"
Jul 17 11:30:27.913549 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:27.913518    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b1f3d1a-3272-4d42-9e15-cfaad6f17ee1" containerName="cleanup"
Jul 17 11:30:27.913549 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:27.913529    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b1f3d1a-3272-4d42-9e15-cfaad6f17ee1" containerName="cleanup"
Jul 17 11:30:27.913549 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:27.913537    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b1f3d1a-3272-4d42-9e15-cfaad6f17ee1" containerName="cleanup"
Jul 17 11:30:27.913850 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:27.913639    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b1f3d1a-3272-4d42-9e15-cfaad6f17ee1" containerName="cleanup"
Jul 17 11:30:27.913850 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:27.913654    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="33863eb2-7000-445d-bfaf-9230a409ef0e" containerName="revoke-keys"
Jul 17 11:30:27.913850 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:27.913664    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b1f3d1a-3272-4d42-9e15-cfaad6f17ee1" containerName="cleanup"
Jul 17 11:30:27.916607 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:27.916587    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-mb5qc"
Jul 17 11:30:27.919736 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:27.919709    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 11:30:27.921531 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:27.921493    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwv2h\""
Jul 17 11:30:27.931454 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:27.931430    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-mb5qc"]
Jul 17 11:30:27.998791 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:27.998754    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/512d8444-195b-43cc-b7a0-8565d015c714-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-40b693f1-mb5qc\" (UID: \"512d8444-195b-43cc-b7a0-8565d015c714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-mb5qc"
Jul 17 11:30:28.099832 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:28.099790    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/512d8444-195b-43cc-b7a0-8565d015c714-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-40b693f1-mb5qc\" (UID: \"512d8444-195b-43cc-b7a0-8565d015c714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-mb5qc"
Jul 17 11:30:28.100432 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:28.100414    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/512d8444-195b-43cc-b7a0-8565d015c714-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-40b693f1-mb5qc\" (UID: \"512d8444-195b-43cc-b7a0-8565d015c714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-mb5qc"
Jul 17 11:30:28.231962 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:28.231877    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-mb5qc"
Jul 17 11:30:28.373879 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:28.373852    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-mb5qc"]
Jul 17 11:30:28.376006 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:30:28.375980    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512d8444_195b_43cc_b7a0_8565d015c714.slice/crio-144ad6f467a186e23e7fc82fff07f955ff9628f10ace711ae44398318ca411e2 WatchSource:0}: Error finding container 144ad6f467a186e23e7fc82fff07f955ff9628f10ace711ae44398318ca411e2: Status 404 returned error can't find the container with id 144ad6f467a186e23e7fc82fff07f955ff9628f10ace711ae44398318ca411e2
Jul 17 11:30:28.390267 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:28.390238    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-mb5qc" event={"ID":"512d8444-195b-43cc-b7a0-8565d015c714","Type":"ContainerStarted","Data":"144ad6f467a186e23e7fc82fff07f955ff9628f10ace711ae44398318ca411e2"}
Jul 17 11:30:29.396832 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:29.396798    2563 generic.go:356] "Generic (PLEG): container finished" podID="512d8444-195b-43cc-b7a0-8565d015c714" containerID="564fd123e62bd2ace73ac1b62c270a44c853052989059af5984e54dec95f465c" exitCode=6
Jul 17 11:30:29.397232 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:29.396860    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-mb5qc" event={"ID":"512d8444-195b-43cc-b7a0-8565d015c714","Type":"ContainerDied","Data":"564fd123e62bd2ace73ac1b62c270a44c853052989059af5984e54dec95f465c"}
Jul 17 11:30:29.397232 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:29.397031    2563 scope.go:117] "RemoveContainer" containerID="564fd123e62bd2ace73ac1b62c270a44c853052989059af5984e54dec95f465c"
Jul 17 11:30:30.403566 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:30.403532    2563 generic.go:356] "Generic (PLEG): container finished" podID="512d8444-195b-43cc-b7a0-8565d015c714" containerID="541a16b13f8649014491bef3cc36ebf61edd5609b90616d66be4400f7bea847c" exitCode=6
Jul 17 11:30:30.404038 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:30.403583    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-mb5qc" event={"ID":"512d8444-195b-43cc-b7a0-8565d015c714","Type":"ContainerDied","Data":"541a16b13f8649014491bef3cc36ebf61edd5609b90616d66be4400f7bea847c"}
Jul 17 11:30:30.404038 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:30.403657    2563 scope.go:117] "RemoveContainer" containerID="564fd123e62bd2ace73ac1b62c270a44c853052989059af5984e54dec95f465c"
Jul 17 11:30:30.404038 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:30.403860    2563 scope.go:117] "RemoveContainer" containerID="541a16b13f8649014491bef3cc36ebf61edd5609b90616d66be4400f7bea847c"
Jul 17 11:30:30.404227 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:30:30.404154    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-40b693f1-mb5qc_odh-ai-gateway-infra(512d8444-195b-43cc-b7a0-8565d015c714)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-mb5qc" podUID="512d8444-195b-43cc-b7a0-8565d015c714"
Jul 17 11:30:31.411081 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:31.410985    2563 scope.go:117] "RemoveContainer" containerID="541a16b13f8649014491bef3cc36ebf61edd5609b90616d66be4400f7bea847c"
Jul 17 11:30:31.411491 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:30:31.411239    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-40b693f1-mb5qc_odh-ai-gateway-infra(512d8444-195b-43cc-b7a0-8565d015c714)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-mb5qc" podUID="512d8444-195b-43cc-b7a0-8565d015c714"
Jul 17 11:30:45.662459 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:45.662423    2563 scope.go:117] "RemoveContainer" containerID="541a16b13f8649014491bef3cc36ebf61edd5609b90616d66be4400f7bea847c"
Jul 17 11:30:46.492414 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:46.492333    2563 generic.go:356] "Generic (PLEG): container finished" podID="512d8444-195b-43cc-b7a0-8565d015c714" containerID="78a27d51eceb8a29eff963aeaa9b747d6a07d2b492f10ff38862b6a92438287c" exitCode=6
Jul 17 11:30:46.492553 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:46.492407    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-mb5qc" event={"ID":"512d8444-195b-43cc-b7a0-8565d015c714","Type":"ContainerDied","Data":"78a27d51eceb8a29eff963aeaa9b747d6a07d2b492f10ff38862b6a92438287c"}
Jul 17 11:30:46.492553 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:46.492463    2563 scope.go:117] "RemoveContainer" containerID="541a16b13f8649014491bef3cc36ebf61edd5609b90616d66be4400f7bea847c"
Jul 17 11:30:46.492761 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:46.492746    2563 scope.go:117] "RemoveContainer" containerID="78a27d51eceb8a29eff963aeaa9b747d6a07d2b492f10ff38862b6a92438287c"
Jul 17 11:30:46.492978 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:30:46.492955    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-40b693f1-mb5qc_odh-ai-gateway-infra(512d8444-195b-43cc-b7a0-8565d015c714)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-mb5qc" podUID="512d8444-195b-43cc-b7a0-8565d015c714"
Jul 17 11:30:47.531103 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:47.531057    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-mb5qc"]
Jul 17 11:30:47.662641 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:47.662616    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-mb5qc"
Jul 17 11:30:47.786233 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:47.786150    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/512d8444-195b-43cc-b7a0-8565d015c714-maas-api-service-ca\") pod \"512d8444-195b-43cc-b7a0-8565d015c714\" (UID: \"512d8444-195b-43cc-b7a0-8565d015c714\") "
Jul 17 11:30:47.786521 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:47.786495    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/512d8444-195b-43cc-b7a0-8565d015c714-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "512d8444-195b-43cc-b7a0-8565d015c714" (UID: "512d8444-195b-43cc-b7a0-8565d015c714"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:30:47.887363 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:47.887328    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/512d8444-195b-43cc-b7a0-8565d015c714-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:30:48.508097 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:48.508048    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-mb5qc"
Jul 17 11:30:48.508097 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:48.508082    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-mb5qc" event={"ID":"512d8444-195b-43cc-b7a0-8565d015c714","Type":"ContainerDied","Data":"144ad6f467a186e23e7fc82fff07f955ff9628f10ace711ae44398318ca411e2"}
Jul 17 11:30:48.508342 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:48.508138    2563 scope.go:117] "RemoveContainer" containerID="78a27d51eceb8a29eff963aeaa9b747d6a07d2b492f10ff38862b6a92438287c"
Jul 17 11:30:48.533274 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:48.533249    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-mb5qc"]
Jul 17 11:30:48.536548 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:48.536527    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-mb5qc"]
Jul 17 11:30:49.665269 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:49.665236    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="512d8444-195b-43cc-b7a0-8565d015c714" path="/var/lib/kubelet/pods/512d8444-195b-43cc-b7a0-8565d015c714/volumes"
Jul 17 11:30:57.984870 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:57.984829    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-hs2fr"]
Jul 17 11:30:57.985466 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:57.985331    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="512d8444-195b-43cc-b7a0-8565d015c714" containerName="revoke-keys"
Jul 17 11:30:57.985466 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:57.985351    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="512d8444-195b-43cc-b7a0-8565d015c714" containerName="revoke-keys"
Jul 17 11:30:57.985466 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:57.985366    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="512d8444-195b-43cc-b7a0-8565d015c714" containerName="revoke-keys"
Jul 17 11:30:57.985466 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:57.985375    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="512d8444-195b-43cc-b7a0-8565d015c714" containerName="revoke-keys"
Jul 17 11:30:57.985677 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:57.985474    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="512d8444-195b-43cc-b7a0-8565d015c714" containerName="revoke-keys"
Jul 17 11:30:57.985677 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:57.985484    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="512d8444-195b-43cc-b7a0-8565d015c714" containerName="revoke-keys"
Jul 17 11:30:57.985677 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:57.985496    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b1f3d1a-3272-4d42-9e15-cfaad6f17ee1" containerName="cleanup"
Jul 17 11:30:57.985677 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:57.985508    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="512d8444-195b-43cc-b7a0-8565d015c714" containerName="revoke-keys"
Jul 17 11:30:57.988508 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:57.988487    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-hs2fr"
Jul 17 11:30:57.991309 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:57.991289    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 11:30:57.991402 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:57.991342    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwv2h\""
Jul 17 11:30:57.997367 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:57.997343    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-hs2fr"]
Jul 17 11:30:58.076423 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:58.076388    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a97ac73-4b1a-4bb1-86ed-c848dd2be79d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-829f05b9-hs2fr\" (UID: \"8a97ac73-4b1a-4bb1-86ed-c848dd2be79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-hs2fr"
Jul 17 11:30:58.177204 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:58.177165    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a97ac73-4b1a-4bb1-86ed-c848dd2be79d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-829f05b9-hs2fr\" (UID: \"8a97ac73-4b1a-4bb1-86ed-c848dd2be79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-hs2fr"
Jul 17 11:30:58.177779 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:58.177756    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a97ac73-4b1a-4bb1-86ed-c848dd2be79d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-829f05b9-hs2fr\" (UID: \"8a97ac73-4b1a-4bb1-86ed-c848dd2be79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-hs2fr"
Jul 17 11:30:58.299933 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:58.299852    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-hs2fr"
Jul 17 11:30:58.432799 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:58.432776    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-hs2fr"]
Jul 17 11:30:58.434895 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:30:58.434864    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a97ac73_4b1a_4bb1_86ed_c848dd2be79d.slice/crio-a84748fb25e6c976b36a07667b075377181fcd57286161ddd63a7a0ae89cb5c9 WatchSource:0}: Error finding container a84748fb25e6c976b36a07667b075377181fcd57286161ddd63a7a0ae89cb5c9: Status 404 returned error can't find the container with id a84748fb25e6c976b36a07667b075377181fcd57286161ddd63a7a0ae89cb5c9
Jul 17 11:30:58.558543 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:58.558457    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-hs2fr" event={"ID":"8a97ac73-4b1a-4bb1-86ed-c848dd2be79d","Type":"ContainerStarted","Data":"a84748fb25e6c976b36a07667b075377181fcd57286161ddd63a7a0ae89cb5c9"}
Jul 17 11:30:59.564763 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:59.564673    2563 generic.go:356] "Generic (PLEG): container finished" podID="8a97ac73-4b1a-4bb1-86ed-c848dd2be79d" containerID="7848afda679992dcc225373fde3ba39906c8a51e106d0405a6db37a4ab1a633c" exitCode=6
Jul 17 11:30:59.565252 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:59.564758    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-hs2fr" event={"ID":"8a97ac73-4b1a-4bb1-86ed-c848dd2be79d","Type":"ContainerDied","Data":"7848afda679992dcc225373fde3ba39906c8a51e106d0405a6db37a4ab1a633c"}
Jul 17 11:30:59.565252 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:30:59.564963    2563 scope.go:117] "RemoveContainer" containerID="7848afda679992dcc225373fde3ba39906c8a51e106d0405a6db37a4ab1a633c"
Jul 17 11:31:00.571498 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:00.571397    2563 generic.go:356] "Generic (PLEG): container finished" podID="8a97ac73-4b1a-4bb1-86ed-c848dd2be79d" containerID="bce302cf63f2b5bc7a3347807a93f0da55a4bc00c7e0853b51047809baf45125" exitCode=6
Jul 17 11:31:00.571498 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:00.571460    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-hs2fr" event={"ID":"8a97ac73-4b1a-4bb1-86ed-c848dd2be79d","Type":"ContainerDied","Data":"bce302cf63f2b5bc7a3347807a93f0da55a4bc00c7e0853b51047809baf45125"}
Jul 17 11:31:00.571959 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:00.571530    2563 scope.go:117] "RemoveContainer" containerID="7848afda679992dcc225373fde3ba39906c8a51e106d0405a6db37a4ab1a633c"
Jul 17 11:31:00.571959 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:00.571683    2563 scope.go:117] "RemoveContainer" containerID="bce302cf63f2b5bc7a3347807a93f0da55a4bc00c7e0853b51047809baf45125"
Jul 17 11:31:00.571959 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:31:00.571901    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-829f05b9-hs2fr_odh-ai-gateway-infra(8a97ac73-4b1a-4bb1-86ed-c848dd2be79d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-hs2fr" podUID="8a97ac73-4b1a-4bb1-86ed-c848dd2be79d"
Jul 17 11:31:01.580649 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:01.580619    2563 scope.go:117] "RemoveContainer" containerID="bce302cf63f2b5bc7a3347807a93f0da55a4bc00c7e0853b51047809baf45125"
Jul 17 11:31:01.581049 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:31:01.580785    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-829f05b9-hs2fr_odh-ai-gateway-infra(8a97ac73-4b1a-4bb1-86ed-c848dd2be79d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-hs2fr" podUID="8a97ac73-4b1a-4bb1-86ed-c848dd2be79d"
Jul 17 11:31:12.660528 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:12.660500    2563 scope.go:117] "RemoveContainer" containerID="bce302cf63f2b5bc7a3347807a93f0da55a4bc00c7e0853b51047809baf45125"
Jul 17 11:31:13.646708 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:13.646674    2563 generic.go:356] "Generic (PLEG): container finished" podID="8a97ac73-4b1a-4bb1-86ed-c848dd2be79d" containerID="4aebf8fb0e89c6db10ef1c463e106833072591077b6e2a86d8cf65fb474894b0" exitCode=6
Jul 17 11:31:13.646874 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:13.646747    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-hs2fr" event={"ID":"8a97ac73-4b1a-4bb1-86ed-c848dd2be79d","Type":"ContainerDied","Data":"4aebf8fb0e89c6db10ef1c463e106833072591077b6e2a86d8cf65fb474894b0"}
Jul 17 11:31:13.646874 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:13.646814    2563 scope.go:117] "RemoveContainer" containerID="bce302cf63f2b5bc7a3347807a93f0da55a4bc00c7e0853b51047809baf45125"
Jul 17 11:31:13.647192 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:13.647158    2563 scope.go:117] "RemoveContainer" containerID="4aebf8fb0e89c6db10ef1c463e106833072591077b6e2a86d8cf65fb474894b0"
Jul 17 11:31:13.647438 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:31:13.647419    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-829f05b9-hs2fr_odh-ai-gateway-infra(8a97ac73-4b1a-4bb1-86ed-c848dd2be79d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-hs2fr" podUID="8a97ac73-4b1a-4bb1-86ed-c848dd2be79d"
Jul 17 11:31:14.676754 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:14.676718    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-hs2fr"]
Jul 17 11:31:14.810551 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:14.810526    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-hs2fr"
Jul 17 11:31:14.925232 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:14.925130    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a97ac73-4b1a-4bb1-86ed-c848dd2be79d-maas-api-service-ca\") pod \"8a97ac73-4b1a-4bb1-86ed-c848dd2be79d\" (UID: \"8a97ac73-4b1a-4bb1-86ed-c848dd2be79d\") "
Jul 17 11:31:14.925528 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:14.925499    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a97ac73-4b1a-4bb1-86ed-c848dd2be79d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8a97ac73-4b1a-4bb1-86ed-c848dd2be79d" (UID: "8a97ac73-4b1a-4bb1-86ed-c848dd2be79d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:31:15.026698 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:15.026613    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a97ac73-4b1a-4bb1-86ed-c848dd2be79d-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:31:15.662952 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:15.662923    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-hs2fr"
Jul 17 11:31:15.665736 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:15.665704    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-hs2fr" event={"ID":"8a97ac73-4b1a-4bb1-86ed-c848dd2be79d","Type":"ContainerDied","Data":"a84748fb25e6c976b36a07667b075377181fcd57286161ddd63a7a0ae89cb5c9"}
Jul 17 11:31:15.665913 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:15.665756    2563 scope.go:117] "RemoveContainer" containerID="4aebf8fb0e89c6db10ef1c463e106833072591077b6e2a86d8cf65fb474894b0"
Jul 17 11:31:15.690342 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:15.690318    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-hs2fr"]
Jul 17 11:31:15.697275 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:15.697255    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-hs2fr"]
Jul 17 11:31:17.664938 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:17.664907    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a97ac73-4b1a-4bb1-86ed-c848dd2be79d" path="/var/lib/kubelet/pods/8a97ac73-4b1a-4bb1-86ed-c848dd2be79d/volumes"
Jul 17 11:31:27.962272 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:27.962236    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp"]
Jul 17 11:31:27.962640 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:27.962573    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a97ac73-4b1a-4bb1-86ed-c848dd2be79d" containerName="revoke-keys"
Jul 17 11:31:27.962640 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:27.962585    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a97ac73-4b1a-4bb1-86ed-c848dd2be79d" containerName="revoke-keys"
Jul 17 11:31:27.962640 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:27.962605    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="512d8444-195b-43cc-b7a0-8565d015c714" containerName="revoke-keys"
Jul 17 11:31:27.962640 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:27.962610    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="512d8444-195b-43cc-b7a0-8565d015c714" containerName="revoke-keys"
Jul 17 11:31:27.962640 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:27.962621    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a97ac73-4b1a-4bb1-86ed-c848dd2be79d" containerName="revoke-keys"
Jul 17 11:31:27.962640 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:27.962626    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a97ac73-4b1a-4bb1-86ed-c848dd2be79d" containerName="revoke-keys"
Jul 17 11:31:27.962835 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:27.962677    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a97ac73-4b1a-4bb1-86ed-c848dd2be79d" containerName="revoke-keys"
Jul 17 11:31:27.962835 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:27.962688    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a97ac73-4b1a-4bb1-86ed-c848dd2be79d" containerName="revoke-keys"
Jul 17 11:31:27.965499 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:27.965482    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp"
Jul 17 11:31:27.968251 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:27.968227    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 11:31:27.968364 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:27.968234    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwv2h\""
Jul 17 11:31:27.976844 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:27.976820    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp"]
Jul 17 11:31:28.143241 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:28.143194    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee32e8ff-7386-4708-8b6e-f50f0f54eb81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp\" (UID: \"ee32e8ff-7386-4708-8b6e-f50f0f54eb81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp"
Jul 17 11:31:28.244147 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:28.244013    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee32e8ff-7386-4708-8b6e-f50f0f54eb81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp\" (UID: \"ee32e8ff-7386-4708-8b6e-f50f0f54eb81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp"
Jul 17 11:31:28.244692 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:28.244668    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee32e8ff-7386-4708-8b6e-f50f0f54eb81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp\" (UID: \"ee32e8ff-7386-4708-8b6e-f50f0f54eb81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp"
Jul 17 11:31:28.275413 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:28.275382    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp"
Jul 17 11:31:28.413569 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:28.413543    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp"]
Jul 17 11:31:28.414551 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:31:28.414521    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee32e8ff_7386_4708_8b6e_f50f0f54eb81.slice/crio-60f86eeae696c0a140b93cdb505db1c4398707681c1e0c8972c4daf9606a7e22 WatchSource:0}: Error finding container 60f86eeae696c0a140b93cdb505db1c4398707681c1e0c8972c4daf9606a7e22: Status 404 returned error can't find the container with id 60f86eeae696c0a140b93cdb505db1c4398707681c1e0c8972c4daf9606a7e22
Jul 17 11:31:28.722769 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:28.722737    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp" event={"ID":"ee32e8ff-7386-4708-8b6e-f50f0f54eb81","Type":"ContainerStarted","Data":"d509bb9a10118a310ad9b489aed9deefbbe3b9df23616d539c04b3014f61667c"}
Jul 17 11:31:28.722916 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:28.722787    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp" event={"ID":"ee32e8ff-7386-4708-8b6e-f50f0f54eb81","Type":"ContainerStarted","Data":"60f86eeae696c0a140b93cdb505db1c4398707681c1e0c8972c4daf9606a7e22"}
Jul 17 11:31:28.738699 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:28.738655    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp" podStartSLOduration=1.738641319 podStartE2EDuration="1.738641319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:31:28.737690366 +0000 UTC m=+2043.667230006" watchObservedRunningTime="2026-07-17 11:31:28.738641319 +0000 UTC m=+2043.668180968"
Jul 17 11:31:29.728756 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:29.728725    2563 generic.go:356] "Generic (PLEG): container finished" podID="ee32e8ff-7386-4708-8b6e-f50f0f54eb81" containerID="d509bb9a10118a310ad9b489aed9deefbbe3b9df23616d539c04b3014f61667c" exitCode=6
Jul 17 11:31:29.729250 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:29.728808    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp" event={"ID":"ee32e8ff-7386-4708-8b6e-f50f0f54eb81","Type":"ContainerDied","Data":"d509bb9a10118a310ad9b489aed9deefbbe3b9df23616d539c04b3014f61667c"}
Jul 17 11:31:29.729250 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:29.729147    2563 scope.go:117] "RemoveContainer" containerID="d509bb9a10118a310ad9b489aed9deefbbe3b9df23616d539c04b3014f61667c"
Jul 17 11:31:30.735526 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:30.735495    2563 generic.go:356] "Generic (PLEG): container finished" podID="ee32e8ff-7386-4708-8b6e-f50f0f54eb81" containerID="d2e4143ef9445da5e0943e9a6bd18b0ab74eb3237f784b3a53f33fb371d2c1df" exitCode=6
Jul 17 11:31:30.735947 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:30.735584    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp" event={"ID":"ee32e8ff-7386-4708-8b6e-f50f0f54eb81","Type":"ContainerDied","Data":"d2e4143ef9445da5e0943e9a6bd18b0ab74eb3237f784b3a53f33fb371d2c1df"}
Jul 17 11:31:30.735947 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:30.735673    2563 scope.go:117] "RemoveContainer" containerID="d509bb9a10118a310ad9b489aed9deefbbe3b9df23616d539c04b3014f61667c"
Jul 17 11:31:30.735947 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:30.735777    2563 scope.go:117] "RemoveContainer" containerID="d2e4143ef9445da5e0943e9a6bd18b0ab74eb3237f784b3a53f33fb371d2c1df"
Jul 17 11:31:30.736139 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:31:30.736008    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp_odh-ai-gateway-infra(ee32e8ff-7386-4708-8b6e-f50f0f54eb81)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp" podUID="ee32e8ff-7386-4708-8b6e-f50f0f54eb81"
Jul 17 11:31:31.746714 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:31.746689    2563 scope.go:117] "RemoveContainer" containerID="d2e4143ef9445da5e0943e9a6bd18b0ab74eb3237f784b3a53f33fb371d2c1df"
Jul 17 11:31:31.747154 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:31:31.746860    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp_odh-ai-gateway-infra(ee32e8ff-7386-4708-8b6e-f50f0f54eb81)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp" podUID="ee32e8ff-7386-4708-8b6e-f50f0f54eb81"
Jul 17 11:31:44.660579 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:44.660549    2563 scope.go:117] "RemoveContainer" containerID="d2e4143ef9445da5e0943e9a6bd18b0ab74eb3237f784b3a53f33fb371d2c1df"
Jul 17 11:31:45.819155 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:45.819118    2563 generic.go:356] "Generic (PLEG): container finished" podID="ee32e8ff-7386-4708-8b6e-f50f0f54eb81" containerID="a319e3926f84949c1815596867a011836e4c436e38782dd11b76079000727cd5" exitCode=6
Jul 17 11:31:45.819633 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:45.819189    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp" event={"ID":"ee32e8ff-7386-4708-8b6e-f50f0f54eb81","Type":"ContainerDied","Data":"a319e3926f84949c1815596867a011836e4c436e38782dd11b76079000727cd5"}
Jul 17 11:31:45.819633 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:45.819248    2563 scope.go:117] "RemoveContainer" containerID="d2e4143ef9445da5e0943e9a6bd18b0ab74eb3237f784b3a53f33fb371d2c1df"
Jul 17 11:31:45.819633 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:45.819581    2563 scope.go:117] "RemoveContainer" containerID="a319e3926f84949c1815596867a011836e4c436e38782dd11b76079000727cd5"
Jul 17 11:31:45.819862 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:31:45.819842    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp_odh-ai-gateway-infra(ee32e8ff-7386-4708-8b6e-f50f0f54eb81)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp" podUID="ee32e8ff-7386-4708-8b6e-f50f0f54eb81"
Jul 17 11:31:46.854024 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:46.853937    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp"]
Jul 17 11:31:46.989739 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:46.989712    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp"
Jul 17 11:31:47.110645 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:47.110553    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee32e8ff-7386-4708-8b6e-f50f0f54eb81-maas-api-service-ca\") pod \"ee32e8ff-7386-4708-8b6e-f50f0f54eb81\" (UID: \"ee32e8ff-7386-4708-8b6e-f50f0f54eb81\") "
Jul 17 11:31:47.110915 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:47.110894    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee32e8ff-7386-4708-8b6e-f50f0f54eb81-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ee32e8ff-7386-4708-8b6e-f50f0f54eb81" (UID: "ee32e8ff-7386-4708-8b6e-f50f0f54eb81"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:31:47.211508 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:47.211473    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee32e8ff-7386-4708-8b6e-f50f0f54eb81-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:31:47.831385 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:47.831349    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp" event={"ID":"ee32e8ff-7386-4708-8b6e-f50f0f54eb81","Type":"ContainerDied","Data":"60f86eeae696c0a140b93cdb505db1c4398707681c1e0c8972c4daf9606a7e22"}
Jul 17 11:31:47.831385 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:47.831372    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp"
Jul 17 11:31:47.831683 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:47.831415    2563 scope.go:117] "RemoveContainer" containerID="a319e3926f84949c1815596867a011836e4c436e38782dd11b76079000727cd5"
Jul 17 11:31:47.856416 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:47.851291    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp"]
Jul 17 11:31:47.860094 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:47.860056    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-x6shp"]
Jul 17 11:31:48.030822 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:48.030748    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs"]
Jul 17 11:31:48.031125 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:48.031111    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee32e8ff-7386-4708-8b6e-f50f0f54eb81" containerName="revoke-keys"
Jul 17 11:31:48.031187 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:48.031127    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee32e8ff-7386-4708-8b6e-f50f0f54eb81" containerName="revoke-keys"
Jul 17 11:31:48.031187 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:48.031136    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a97ac73-4b1a-4bb1-86ed-c848dd2be79d" containerName="revoke-keys"
Jul 17 11:31:48.031187 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:48.031142    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a97ac73-4b1a-4bb1-86ed-c848dd2be79d" containerName="revoke-keys"
Jul 17 11:31:48.031187 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:48.031170    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee32e8ff-7386-4708-8b6e-f50f0f54eb81" containerName="revoke-keys"
Jul 17 11:31:48.031187 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:48.031177    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee32e8ff-7386-4708-8b6e-f50f0f54eb81" containerName="revoke-keys"
Jul 17 11:31:48.031187 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:48.031183    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee32e8ff-7386-4708-8b6e-f50f0f54eb81" containerName="revoke-keys"
Jul 17 11:31:48.031187 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:48.031188    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee32e8ff-7386-4708-8b6e-f50f0f54eb81" containerName="revoke-keys"
Jul 17 11:31:48.031390 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:48.031244    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a97ac73-4b1a-4bb1-86ed-c848dd2be79d" containerName="revoke-keys"
Jul 17 11:31:48.031390 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:48.031250    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee32e8ff-7386-4708-8b6e-f50f0f54eb81" containerName="revoke-keys"
Jul 17 11:31:48.031390 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:48.031257    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee32e8ff-7386-4708-8b6e-f50f0f54eb81" containerName="revoke-keys"
Jul 17 11:31:48.031390 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:48.031262    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee32e8ff-7386-4708-8b6e-f50f0f54eb81" containerName="revoke-keys"
Jul 17 11:31:48.033958 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:48.033942    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs"
Jul 17 11:31:48.036391 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:48.036370    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwv2h\""
Jul 17 11:31:48.036509 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:48.036481    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 11:31:48.045328 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:48.045300    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs"]
Jul 17 11:31:48.120246 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:48.120195    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74093ec8-b93d-4da8-bad0-98a151c5d9e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs\" (UID: \"74093ec8-b93d-4da8-bad0-98a151c5d9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs"
Jul 17 11:31:48.221014 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:48.220969    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74093ec8-b93d-4da8-bad0-98a151c5d9e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs\" (UID: \"74093ec8-b93d-4da8-bad0-98a151c5d9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs"
Jul 17 11:31:48.221674 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:48.221646    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74093ec8-b93d-4da8-bad0-98a151c5d9e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs\" (UID: \"74093ec8-b93d-4da8-bad0-98a151c5d9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs"
Jul 17 11:31:48.343982 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:48.343897    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs"
Jul 17 11:31:48.474748 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:48.474716    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs"]
Jul 17 11:31:48.477012 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:31:48.476985    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74093ec8_b93d_4da8_bad0_98a151c5d9e8.slice/crio-e94f4d71fdc396772dfbc899c085d74690dce08c656604fd6adf2a6e593d6d09 WatchSource:0}: Error finding container e94f4d71fdc396772dfbc899c085d74690dce08c656604fd6adf2a6e593d6d09: Status 404 returned error can't find the container with id e94f4d71fdc396772dfbc899c085d74690dce08c656604fd6adf2a6e593d6d09
Jul 17 11:31:48.838396 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:48.838360    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs" event={"ID":"74093ec8-b93d-4da8-bad0-98a151c5d9e8","Type":"ContainerStarted","Data":"762120700beff97412362383f5091aaa4a3975588e631998e5f7c1f591a3858e"}
Jul 17 11:31:48.838576 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:48.838421    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs" event={"ID":"74093ec8-b93d-4da8-bad0-98a151c5d9e8","Type":"ContainerStarted","Data":"e94f4d71fdc396772dfbc899c085d74690dce08c656604fd6adf2a6e593d6d09"}
Jul 17 11:31:48.857424 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:48.857376    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs" podStartSLOduration=0.857360635 podStartE2EDuration="857.360635ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:31:48.85503947 +0000 UTC m=+2063.784579135" watchObservedRunningTime="2026-07-17 11:31:48.857360635 +0000 UTC m=+2063.786900276"
Jul 17 11:31:49.664660 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:49.664626    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee32e8ff-7386-4708-8b6e-f50f0f54eb81" path="/var/lib/kubelet/pods/ee32e8ff-7386-4708-8b6e-f50f0f54eb81/volumes"
Jul 17 11:31:49.845818 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:49.845784    2563 generic.go:356] "Generic (PLEG): container finished" podID="74093ec8-b93d-4da8-bad0-98a151c5d9e8" containerID="762120700beff97412362383f5091aaa4a3975588e631998e5f7c1f591a3858e" exitCode=6
Jul 17 11:31:49.846018 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:49.845834    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs" event={"ID":"74093ec8-b93d-4da8-bad0-98a151c5d9e8","Type":"ContainerDied","Data":"762120700beff97412362383f5091aaa4a3975588e631998e5f7c1f591a3858e"}
Jul 17 11:31:49.846148 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:49.846134    2563 scope.go:117] "RemoveContainer" containerID="762120700beff97412362383f5091aaa4a3975588e631998e5f7c1f591a3858e"
Jul 17 11:31:50.359128 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:50.359047    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:31:50.852047 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:50.852010    2563 generic.go:356] "Generic (PLEG): container finished" podID="74093ec8-b93d-4da8-bad0-98a151c5d9e8" containerID="c2f50a4441fdf6ca2eb770ba1c55910808586efa12136d560651bcf3a7f915eb" exitCode=6
Jul 17 11:31:50.852229 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:50.852081    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs" event={"ID":"74093ec8-b93d-4da8-bad0-98a151c5d9e8","Type":"ContainerDied","Data":"c2f50a4441fdf6ca2eb770ba1c55910808586efa12136d560651bcf3a7f915eb"}
Jul 17 11:31:50.852229 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:50.852141    2563 scope.go:117] "RemoveContainer" containerID="762120700beff97412362383f5091aaa4a3975588e631998e5f7c1f591a3858e"
Jul 17 11:31:50.852346 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:50.852326    2563 scope.go:117] "RemoveContainer" containerID="c2f50a4441fdf6ca2eb770ba1c55910808586efa12136d560651bcf3a7f915eb"
Jul 17 11:31:50.852561 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:31:50.852539    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs_odh-ai-gateway-infra(74093ec8-b93d-4da8-bad0-98a151c5d9e8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs" podUID="74093ec8-b93d-4da8-bad0-98a151c5d9e8"
Jul 17 11:31:51.642587 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:51.642555    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:31:51.858856 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:31:51.858826    2563 scope.go:117] "RemoveContainer" containerID="c2f50a4441fdf6ca2eb770ba1c55910808586efa12136d560651bcf3a7f915eb"
Jul 17 11:31:51.859061 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:31:51.859033    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs_odh-ai-gateway-infra(74093ec8-b93d-4da8-bad0-98a151c5d9e8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs" podUID="74093ec8-b93d-4da8-bad0-98a151c5d9e8"
Jul 17 11:32:02.660175 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:02.660141    2563 scope.go:117] "RemoveContainer" containerID="c2f50a4441fdf6ca2eb770ba1c55910808586efa12136d560651bcf3a7f915eb"
Jul 17 11:32:03.921733 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:03.921701    2563 generic.go:356] "Generic (PLEG): container finished" podID="74093ec8-b93d-4da8-bad0-98a151c5d9e8" containerID="6871e5014d11c70c4989b97bb1434dd0aa8d6a4939d57a58e233d156a556d827" exitCode=6
Jul 17 11:32:03.922152 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:03.921740    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs" event={"ID":"74093ec8-b93d-4da8-bad0-98a151c5d9e8","Type":"ContainerDied","Data":"6871e5014d11c70c4989b97bb1434dd0aa8d6a4939d57a58e233d156a556d827"}
Jul 17 11:32:03.922152 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:03.921791    2563 scope.go:117] "RemoveContainer" containerID="c2f50a4441fdf6ca2eb770ba1c55910808586efa12136d560651bcf3a7f915eb"
Jul 17 11:32:03.922227 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:03.922206    2563 scope.go:117] "RemoveContainer" containerID="6871e5014d11c70c4989b97bb1434dd0aa8d6a4939d57a58e233d156a556d827"
Jul 17 11:32:03.922445 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:32:03.922417    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs_odh-ai-gateway-infra(74093ec8-b93d-4da8-bad0-98a151c5d9e8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs" podUID="74093ec8-b93d-4da8-bad0-98a151c5d9e8"
Jul 17 11:32:04.952084 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:04.952042    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs"]
Jul 17 11:32:05.105256 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:05.105227    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs"
Jul 17 11:32:05.173309 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:05.173280    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74093ec8-b93d-4da8-bad0-98a151c5d9e8-maas-api-service-ca\") pod \"74093ec8-b93d-4da8-bad0-98a151c5d9e8\" (UID: \"74093ec8-b93d-4da8-bad0-98a151c5d9e8\") "
Jul 17 11:32:05.173685 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:05.173662    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74093ec8-b93d-4da8-bad0-98a151c5d9e8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "74093ec8-b93d-4da8-bad0-98a151c5d9e8" (UID: "74093ec8-b93d-4da8-bad0-98a151c5d9e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:32:05.274483 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:05.274389    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74093ec8-b93d-4da8-bad0-98a151c5d9e8-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:32:05.935164 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:05.935052    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs"
Jul 17 11:32:05.935164 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:05.935051    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs" event={"ID":"74093ec8-b93d-4da8-bad0-98a151c5d9e8","Type":"ContainerDied","Data":"e94f4d71fdc396772dfbc899c085d74690dce08c656604fd6adf2a6e593d6d09"}
Jul 17 11:32:05.935164 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:05.935160    2563 scope.go:117] "RemoveContainer" containerID="6871e5014d11c70c4989b97bb1434dd0aa8d6a4939d57a58e233d156a556d827"
Jul 17 11:32:05.957282 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:05.957255    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs"]
Jul 17 11:32:05.962948 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:05.962924    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-fgxqs"]
Jul 17 11:32:07.624251 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:07.624217    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:32:07.666285 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:07.666256    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74093ec8-b93d-4da8-bad0-98a151c5d9e8" path="/var/lib/kubelet/pods/74093ec8-b93d-4da8-bad0-98a151c5d9e8/volumes"
Jul 17 11:32:07.725280 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:07.725243    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:32:18.021524 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:18.021492    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w"]
Jul 17 11:32:18.021990 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:18.021814    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74093ec8-b93d-4da8-bad0-98a151c5d9e8" containerName="revoke-keys"
Jul 17 11:32:18.021990 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:18.021825    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="74093ec8-b93d-4da8-bad0-98a151c5d9e8" containerName="revoke-keys"
Jul 17 11:32:18.021990 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:18.021835    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74093ec8-b93d-4da8-bad0-98a151c5d9e8" containerName="revoke-keys"
Jul 17 11:32:18.021990 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:18.021840    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="74093ec8-b93d-4da8-bad0-98a151c5d9e8" containerName="revoke-keys"
Jul 17 11:32:18.021990 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:18.021852    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74093ec8-b93d-4da8-bad0-98a151c5d9e8" containerName="revoke-keys"
Jul 17 11:32:18.021990 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:18.021857    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="74093ec8-b93d-4da8-bad0-98a151c5d9e8" containerName="revoke-keys"
Jul 17 11:32:18.021990 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:18.021928    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="74093ec8-b93d-4da8-bad0-98a151c5d9e8" containerName="revoke-keys"
Jul 17 11:32:18.021990 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:18.021939    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="74093ec8-b93d-4da8-bad0-98a151c5d9e8" containerName="revoke-keys"
Jul 17 11:32:18.026225 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:18.026208    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w"
Jul 17 11:32:18.029047 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:18.029024    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwv2h\""
Jul 17 11:32:18.029889 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:18.029874    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 11:32:18.038025 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:18.038000    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w"]
Jul 17 11:32:18.091149 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:18.091120    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b350157c-c073-4800-b6db-4c521034a119-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w\" (UID: \"b350157c-c073-4800-b6db-4c521034a119\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w"
Jul 17 11:32:18.192027 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:18.191985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b350157c-c073-4800-b6db-4c521034a119-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w\" (UID: \"b350157c-c073-4800-b6db-4c521034a119\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w"
Jul 17 11:32:18.192607 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:18.192586    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b350157c-c073-4800-b6db-4c521034a119-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w\" (UID: \"b350157c-c073-4800-b6db-4c521034a119\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w"
Jul 17 11:32:18.336261 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:18.336180    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w"
Jul 17 11:32:18.470784 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:18.470756    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w"]
Jul 17 11:32:18.472492 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:32:18.472468    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb350157c_c073_4800_b6db_4c521034a119.slice/crio-99b0e38ea76227e2442bbbe5496a06d7443857df30af5efc61bb342f95098074 WatchSource:0}: Error finding container 99b0e38ea76227e2442bbbe5496a06d7443857df30af5efc61bb342f95098074: Status 404 returned error can't find the container with id 99b0e38ea76227e2442bbbe5496a06d7443857df30af5efc61bb342f95098074
Jul 17 11:32:19.000551 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:19.000460    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w" event={"ID":"b350157c-c073-4800-b6db-4c521034a119","Type":"ContainerStarted","Data":"dcc9843e0eac590cf135f4fbe984514aba427a335db1e679a64be72a5b11eec8"}
Jul 17 11:32:19.000551 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:19.000519    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w" event={"ID":"b350157c-c073-4800-b6db-4c521034a119","Type":"ContainerStarted","Data":"99b0e38ea76227e2442bbbe5496a06d7443857df30af5efc61bb342f95098074"}
Jul 17 11:32:19.022599 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:19.022549    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w" podStartSLOduration=1.022535454 podStartE2EDuration="1.022535454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:32:19.019533799 +0000 UTC m=+2093.949073452" watchObservedRunningTime="2026-07-17 11:32:19.022535454 +0000 UTC m=+2093.952075095"
Jul 17 11:32:20.006757 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:20.006717    2563 generic.go:356] "Generic (PLEG): container finished" podID="b350157c-c073-4800-b6db-4c521034a119" containerID="dcc9843e0eac590cf135f4fbe984514aba427a335db1e679a64be72a5b11eec8" exitCode=6
Jul 17 11:32:20.006977 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:20.006756    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w" event={"ID":"b350157c-c073-4800-b6db-4c521034a119","Type":"ContainerDied","Data":"dcc9843e0eac590cf135f4fbe984514aba427a335db1e679a64be72a5b11eec8"}
Jul 17 11:32:20.007050 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:20.007007    2563 scope.go:117] "RemoveContainer" containerID="dcc9843e0eac590cf135f4fbe984514aba427a335db1e679a64be72a5b11eec8"
Jul 17 11:32:21.017410 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:21.017376    2563 generic.go:356] "Generic (PLEG): container finished" podID="b350157c-c073-4800-b6db-4c521034a119" containerID="e2884b927fc192fdb09c41f62864cee475c3959ac9da4289415a7808b1d85960" exitCode=6
Jul 17 11:32:21.017817 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:21.017435    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w" event={"ID":"b350157c-c073-4800-b6db-4c521034a119","Type":"ContainerDied","Data":"e2884b927fc192fdb09c41f62864cee475c3959ac9da4289415a7808b1d85960"}
Jul 17 11:32:21.017817 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:21.017486    2563 scope.go:117] "RemoveContainer" containerID="dcc9843e0eac590cf135f4fbe984514aba427a335db1e679a64be72a5b11eec8"
Jul 17 11:32:21.017817 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:21.017633    2563 scope.go:117] "RemoveContainer" containerID="e2884b927fc192fdb09c41f62864cee475c3959ac9da4289415a7808b1d85960"
Jul 17 11:32:21.017936 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:32:21.017881    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w_odh-ai-gateway-infra(b350157c-c073-4800-b6db-4c521034a119)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w" podUID="b350157c-c073-4800-b6db-4c521034a119"
Jul 17 11:32:22.024570 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:22.024539    2563 scope.go:117] "RemoveContainer" containerID="e2884b927fc192fdb09c41f62864cee475c3959ac9da4289415a7808b1d85960"
Jul 17 11:32:22.024960 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:32:22.024725    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w_odh-ai-gateway-infra(b350157c-c073-4800-b6db-4c521034a119)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w" podUID="b350157c-c073-4800-b6db-4c521034a119"
Jul 17 11:32:25.698293 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:25.698259    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nj5mj_55d1de37-8635-4de8-ab59-4ebfd0346a40/ovn-acl-logging/0.log"
Jul 17 11:32:25.705451 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:25.705431    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nj5mj_55d1de37-8635-4de8-ab59-4ebfd0346a40/ovn-acl-logging/0.log"
Jul 17 11:32:34.660008 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:34.659977    2563 scope.go:117] "RemoveContainer" containerID="e2884b927fc192fdb09c41f62864cee475c3959ac9da4289415a7808b1d85960"
Jul 17 11:32:35.092519 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:35.092485    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w" event={"ID":"b350157c-c073-4800-b6db-4c521034a119","Type":"ContainerStarted","Data":"7310269f036ef4d1f3a9641fe35cc569c915210618dc44c8cd63a16bfc49d1a3"}
Jul 17 11:32:36.098653 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:36.098612    2563 generic.go:356] "Generic (PLEG): container finished" podID="b350157c-c073-4800-b6db-4c521034a119" containerID="7310269f036ef4d1f3a9641fe35cc569c915210618dc44c8cd63a16bfc49d1a3" exitCode=6
Jul 17 11:32:36.099063 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:36.098684    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w" event={"ID":"b350157c-c073-4800-b6db-4c521034a119","Type":"ContainerDied","Data":"7310269f036ef4d1f3a9641fe35cc569c915210618dc44c8cd63a16bfc49d1a3"}
Jul 17 11:32:36.099063 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:36.098747    2563 scope.go:117] "RemoveContainer" containerID="e2884b927fc192fdb09c41f62864cee475c3959ac9da4289415a7808b1d85960"
Jul 17 11:32:36.099063 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:36.098973    2563 scope.go:117] "RemoveContainer" containerID="7310269f036ef4d1f3a9641fe35cc569c915210618dc44c8cd63a16bfc49d1a3"
Jul 17 11:32:36.099223 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:32:36.099196    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w_odh-ai-gateway-infra(b350157c-c073-4800-b6db-4c521034a119)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w" podUID="b350157c-c073-4800-b6db-4c521034a119"
Jul 17 11:32:36.127788 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:36.127761    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w"]
Jul 17 11:32:37.232550 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:37.232529    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w"
Jul 17 11:32:37.362287 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:37.362202    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b350157c-c073-4800-b6db-4c521034a119-maas-api-service-ca\") pod \"b350157c-c073-4800-b6db-4c521034a119\" (UID: \"b350157c-c073-4800-b6db-4c521034a119\") "
Jul 17 11:32:37.362574 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:37.362554    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b350157c-c073-4800-b6db-4c521034a119-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b350157c-c073-4800-b6db-4c521034a119" (UID: "b350157c-c073-4800-b6db-4c521034a119"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:32:37.462813 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:37.462778    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b350157c-c073-4800-b6db-4c521034a119-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:32:38.080966 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:38.080881    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-45nsx"]
Jul 17 11:32:38.081260 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:38.081246    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b350157c-c073-4800-b6db-4c521034a119" containerName="revoke-keys"
Jul 17 11:32:38.081306 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:38.081261    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b350157c-c073-4800-b6db-4c521034a119" containerName="revoke-keys"
Jul 17 11:32:38.081306 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:38.081276    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b350157c-c073-4800-b6db-4c521034a119" containerName="revoke-keys"
Jul 17 11:32:38.081306 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:38.081282    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b350157c-c073-4800-b6db-4c521034a119" containerName="revoke-keys"
Jul 17 11:32:38.081393 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:38.081327    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b350157c-c073-4800-b6db-4c521034a119" containerName="revoke-keys"
Jul 17 11:32:38.081393 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:38.081336    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b350157c-c073-4800-b6db-4c521034a119" containerName="revoke-keys"
Jul 17 11:32:38.081393 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:38.081344    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="74093ec8-b93d-4da8-bad0-98a151c5d9e8" containerName="revoke-keys"
Jul 17 11:32:38.084140 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:38.084125    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-45nsx"
Jul 17 11:32:38.094580 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:38.094556    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-45nsx"]
Jul 17 11:32:38.111370 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:38.111340    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w" event={"ID":"b350157c-c073-4800-b6db-4c521034a119","Type":"ContainerDied","Data":"99b0e38ea76227e2442bbbe5496a06d7443857df30af5efc61bb342f95098074"}
Jul 17 11:32:38.111370 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:38.111364    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w"
Jul 17 11:32:38.111589 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:38.111396    2563 scope.go:117] "RemoveContainer" containerID="7310269f036ef4d1f3a9641fe35cc569c915210618dc44c8cd63a16bfc49d1a3"
Jul 17 11:32:38.131756 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:38.131730    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w"]
Jul 17 11:32:38.140693 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:38.140660    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-dll6w"]
Jul 17 11:32:38.168716 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:38.168696    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41224b99-2f2d-45a7-b37c-5d2a59fd2323-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-829f05b9-45nsx\" (UID: \"41224b99-2f2d-45a7-b37c-5d2a59fd2323\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-45nsx"
Jul 17 11:32:38.270143 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:38.270114    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41224b99-2f2d-45a7-b37c-5d2a59fd2323-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-829f05b9-45nsx\" (UID: \"41224b99-2f2d-45a7-b37c-5d2a59fd2323\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-45nsx"
Jul 17 11:32:38.271257 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:38.271229    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41224b99-2f2d-45a7-b37c-5d2a59fd2323-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-829f05b9-45nsx\" (UID: \"41224b99-2f2d-45a7-b37c-5d2a59fd2323\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-45nsx"
Jul 17 11:32:38.395733 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:38.395703    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-45nsx"
Jul 17 11:32:38.529417 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:38.529389    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-45nsx"]
Jul 17 11:32:38.531788 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:32:38.531760    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41224b99_2f2d_45a7_b37c_5d2a59fd2323.slice/crio-e0cd62cef81d75d6ef52e2e5dfd015916bc141d9a1ddcf865b322e442cfc3cd5 WatchSource:0}: Error finding container e0cd62cef81d75d6ef52e2e5dfd015916bc141d9a1ddcf865b322e442cfc3cd5: Status 404 returned error can't find the container with id e0cd62cef81d75d6ef52e2e5dfd015916bc141d9a1ddcf865b322e442cfc3cd5
Jul 17 11:32:39.119308 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:39.119211    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-45nsx" event={"ID":"41224b99-2f2d-45a7-b37c-5d2a59fd2323","Type":"ContainerStarted","Data":"4f6d766c06c998e2b76d586e01db9564d9a9fa5af99b86b6f3cb3c8d18b02e71"}
Jul 17 11:32:39.119308 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:39.119269    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-45nsx" event={"ID":"41224b99-2f2d-45a7-b37c-5d2a59fd2323","Type":"ContainerStarted","Data":"e0cd62cef81d75d6ef52e2e5dfd015916bc141d9a1ddcf865b322e442cfc3cd5"}
Jul 17 11:32:39.138439 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:39.138385    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-45nsx" podStartSLOduration=1.138370661 podStartE2EDuration="1.138370661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:32:39.137592823 +0000 UTC m=+2114.067132454" watchObservedRunningTime="2026-07-17 11:32:39.138370661 +0000 UTC m=+2114.067910302"
Jul 17 11:32:39.666972 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:39.666937    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b350157c-c073-4800-b6db-4c521034a119" path="/var/lib/kubelet/pods/b350157c-c073-4800-b6db-4c521034a119/volumes"
Jul 17 11:32:40.125861 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:40.125775    2563 generic.go:356] "Generic (PLEG): container finished" podID="41224b99-2f2d-45a7-b37c-5d2a59fd2323" containerID="4f6d766c06c998e2b76d586e01db9564d9a9fa5af99b86b6f3cb3c8d18b02e71" exitCode=6
Jul 17 11:32:40.126045 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:40.125864    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-45nsx" event={"ID":"41224b99-2f2d-45a7-b37c-5d2a59fd2323","Type":"ContainerDied","Data":"4f6d766c06c998e2b76d586e01db9564d9a9fa5af99b86b6f3cb3c8d18b02e71"}
Jul 17 11:32:40.126124 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:40.126060    2563 scope.go:117] "RemoveContainer" containerID="4f6d766c06c998e2b76d586e01db9564d9a9fa5af99b86b6f3cb3c8d18b02e71"
Jul 17 11:32:41.135677 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:41.135641    2563 generic.go:356] "Generic (PLEG): container finished" podID="41224b99-2f2d-45a7-b37c-5d2a59fd2323" containerID="4559e2310ecf8b06072aaf5af89b67654fdb4fdeb87af89587b8e692763590d5" exitCode=6
Jul 17 11:32:41.136089 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:41.135722    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-45nsx" event={"ID":"41224b99-2f2d-45a7-b37c-5d2a59fd2323","Type":"ContainerDied","Data":"4559e2310ecf8b06072aaf5af89b67654fdb4fdeb87af89587b8e692763590d5"}
Jul 17 11:32:41.136089 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:41.135789    2563 scope.go:117] "RemoveContainer" containerID="4f6d766c06c998e2b76d586e01db9564d9a9fa5af99b86b6f3cb3c8d18b02e71"
Jul 17 11:32:41.136089 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:41.136020    2563 scope.go:117] "RemoveContainer" containerID="4559e2310ecf8b06072aaf5af89b67654fdb4fdeb87af89587b8e692763590d5"
Jul 17 11:32:41.136266 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:32:41.136250    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-829f05b9-45nsx_odh-ai-gateway-infra(41224b99-2f2d-45a7-b37c-5d2a59fd2323)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-45nsx" podUID="41224b99-2f2d-45a7-b37c-5d2a59fd2323"
Jul 17 11:32:42.142777 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:42.142748    2563 scope.go:117] "RemoveContainer" containerID="4559e2310ecf8b06072aaf5af89b67654fdb4fdeb87af89587b8e692763590d5"
Jul 17 11:32:42.143181 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:32:42.142960    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-829f05b9-45nsx_odh-ai-gateway-infra(41224b99-2f2d-45a7-b37c-5d2a59fd2323)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-45nsx" podUID="41224b99-2f2d-45a7-b37c-5d2a59fd2323"
Jul 17 11:32:55.662215 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:55.662176    2563 scope.go:117] "RemoveContainer" containerID="4559e2310ecf8b06072aaf5af89b67654fdb4fdeb87af89587b8e692763590d5"
Jul 17 11:32:56.210877 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:56.210844    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-45nsx" event={"ID":"41224b99-2f2d-45a7-b37c-5d2a59fd2323","Type":"ContainerStarted","Data":"85f39ee2707a664063d877896934c949f3fb516719b9f45d43c70930c37d26b9"}
Jul 17 11:32:57.217138 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:57.217106    2563 generic.go:356] "Generic (PLEG): container finished" podID="41224b99-2f2d-45a7-b37c-5d2a59fd2323" containerID="85f39ee2707a664063d877896934c949f3fb516719b9f45d43c70930c37d26b9" exitCode=6
Jul 17 11:32:57.217629 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:57.217163    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-45nsx" event={"ID":"41224b99-2f2d-45a7-b37c-5d2a59fd2323","Type":"ContainerDied","Data":"85f39ee2707a664063d877896934c949f3fb516719b9f45d43c70930c37d26b9"}
Jul 17 11:32:57.217629 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:57.217211    2563 scope.go:117] "RemoveContainer" containerID="4559e2310ecf8b06072aaf5af89b67654fdb4fdeb87af89587b8e692763590d5"
Jul 17 11:32:57.217629 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:57.217454    2563 scope.go:117] "RemoveContainer" containerID="85f39ee2707a664063d877896934c949f3fb516719b9f45d43c70930c37d26b9"
Jul 17 11:32:57.217746 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:32:57.217682    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-829f05b9-45nsx_odh-ai-gateway-infra(41224b99-2f2d-45a7-b37c-5d2a59fd2323)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-45nsx" podUID="41224b99-2f2d-45a7-b37c-5d2a59fd2323"
Jul 17 11:32:57.249234 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:57.249204    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-45nsx"]
Jul 17 11:32:58.352571 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:58.352548    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-45nsx"
Jul 17 11:32:58.454062 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:58.454026    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41224b99-2f2d-45a7-b37c-5d2a59fd2323-maas-api-service-ca\") pod \"41224b99-2f2d-45a7-b37c-5d2a59fd2323\" (UID: \"41224b99-2f2d-45a7-b37c-5d2a59fd2323\") "
Jul 17 11:32:58.454432 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:58.454406    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41224b99-2f2d-45a7-b37c-5d2a59fd2323-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "41224b99-2f2d-45a7-b37c-5d2a59fd2323" (UID: "41224b99-2f2d-45a7-b37c-5d2a59fd2323"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:32:58.555450 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:58.555363    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41224b99-2f2d-45a7-b37c-5d2a59fd2323-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:32:59.229696 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:59.229658    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-45nsx" event={"ID":"41224b99-2f2d-45a7-b37c-5d2a59fd2323","Type":"ContainerDied","Data":"e0cd62cef81d75d6ef52e2e5dfd015916bc141d9a1ddcf865b322e442cfc3cd5"}
Jul 17 11:32:59.229880 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:59.229716    2563 scope.go:117] "RemoveContainer" containerID="85f39ee2707a664063d877896934c949f3fb516719b9f45d43c70930c37d26b9"
Jul 17 11:32:59.229880 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:59.229673    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-45nsx"
Jul 17 11:32:59.262385 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:59.262360    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-45nsx"]
Jul 17 11:32:59.269296 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:59.266060    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-45nsx"]
Jul 17 11:32:59.665544 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:32:59.665503    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41224b99-2f2d-45a7-b37c-5d2a59fd2323" path="/var/lib/kubelet/pods/41224b99-2f2d-45a7-b37c-5d2a59fd2323/volumes"
Jul 17 11:33:26.987980 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:26.987945    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-f5p2q"]
Jul 17 11:33:26.988336 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:26.988273    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41224b99-2f2d-45a7-b37c-5d2a59fd2323" containerName="revoke-keys"
Jul 17 11:33:26.988336 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:26.988284    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="41224b99-2f2d-45a7-b37c-5d2a59fd2323" containerName="revoke-keys"
Jul 17 11:33:26.988336 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:26.988296    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41224b99-2f2d-45a7-b37c-5d2a59fd2323" containerName="revoke-keys"
Jul 17 11:33:26.988336 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:26.988301    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="41224b99-2f2d-45a7-b37c-5d2a59fd2323" containerName="revoke-keys"
Jul 17 11:33:26.988336 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:26.988314    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41224b99-2f2d-45a7-b37c-5d2a59fd2323" containerName="revoke-keys"
Jul 17 11:33:26.988336 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:26.988320    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="41224b99-2f2d-45a7-b37c-5d2a59fd2323" containerName="revoke-keys"
Jul 17 11:33:26.988336 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:26.988327    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b350157c-c073-4800-b6db-4c521034a119" containerName="revoke-keys"
Jul 17 11:33:26.988336 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:26.988332    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b350157c-c073-4800-b6db-4c521034a119" containerName="revoke-keys"
Jul 17 11:33:26.988592 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:26.988384    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="41224b99-2f2d-45a7-b37c-5d2a59fd2323" containerName="revoke-keys"
Jul 17 11:33:26.988592 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:26.988392    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="41224b99-2f2d-45a7-b37c-5d2a59fd2323" containerName="revoke-keys"
Jul 17 11:33:26.988592 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:26.988398    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b350157c-c073-4800-b6db-4c521034a119" containerName="revoke-keys"
Jul 17 11:33:26.992740 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:26.992724    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-f5p2q"
Jul 17 11:33:26.995952 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:26.995501    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwv2h\""
Jul 17 11:33:26.995952 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:26.995498    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 11:33:26.999922 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:26.999900    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-f5p2q"]
Jul 17 11:33:27.105651 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:27.105599    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a21f1f-570d-4aef-af72-f8487da613e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45989be5-f5p2q\" (UID: \"61a21f1f-570d-4aef-af72-f8487da613e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-f5p2q"
Jul 17 11:33:27.206877 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:27.206840    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a21f1f-570d-4aef-af72-f8487da613e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45989be5-f5p2q\" (UID: \"61a21f1f-570d-4aef-af72-f8487da613e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-f5p2q"
Jul 17 11:33:27.207508 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:27.207490    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a21f1f-570d-4aef-af72-f8487da613e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45989be5-f5p2q\" (UID: \"61a21f1f-570d-4aef-af72-f8487da613e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-f5p2q"
Jul 17 11:33:27.304974 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:27.304898    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-f5p2q"
Jul 17 11:33:27.434198 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:27.434172    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-f5p2q"]
Jul 17 11:33:27.435838 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:33:27.435814    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a21f1f_570d_4aef_af72_f8487da613e7.slice/crio-ca22c05b262383fc90598852e9a383257fb3cf29591297e9af4b08905f1e67ee WatchSource:0}: Error finding container ca22c05b262383fc90598852e9a383257fb3cf29591297e9af4b08905f1e67ee: Status 404 returned error can't find the container with id ca22c05b262383fc90598852e9a383257fb3cf29591297e9af4b08905f1e67ee
Jul 17 11:33:28.380353 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:28.380275    2563 generic.go:356] "Generic (PLEG): container finished" podID="61a21f1f-570d-4aef-af72-f8487da613e7" containerID="5fee92f988121fc718fd0b292261ece37cd8c6ec00ce9cfc626e6326f413851d" exitCode=7
Jul 17 11:33:28.380766 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:28.380359    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-f5p2q" event={"ID":"61a21f1f-570d-4aef-af72-f8487da613e7","Type":"ContainerDied","Data":"5fee92f988121fc718fd0b292261ece37cd8c6ec00ce9cfc626e6326f413851d"}
Jul 17 11:33:28.380766 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:28.380422    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-f5p2q" event={"ID":"61a21f1f-570d-4aef-af72-f8487da613e7","Type":"ContainerStarted","Data":"ca22c05b262383fc90598852e9a383257fb3cf29591297e9af4b08905f1e67ee"}
Jul 17 11:33:28.380766 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:28.380523    2563 scope.go:117] "RemoveContainer" containerID="5fee92f988121fc718fd0b292261ece37cd8c6ec00ce9cfc626e6326f413851d"
Jul 17 11:33:29.389275 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:29.389240    2563 generic.go:356] "Generic (PLEG): container finished" podID="61a21f1f-570d-4aef-af72-f8487da613e7" containerID="14d89b10125d768376563ffab39601050f85e96e2de5aef894b11e475dd85a25" exitCode=7
Jul 17 11:33:29.389689 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:29.389327    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-f5p2q" event={"ID":"61a21f1f-570d-4aef-af72-f8487da613e7","Type":"ContainerDied","Data":"14d89b10125d768376563ffab39601050f85e96e2de5aef894b11e475dd85a25"}
Jul 17 11:33:29.389689 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:29.389381    2563 scope.go:117] "RemoveContainer" containerID="5fee92f988121fc718fd0b292261ece37cd8c6ec00ce9cfc626e6326f413851d"
Jul 17 11:33:29.389689 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:29.389532    2563 scope.go:117] "RemoveContainer" containerID="14d89b10125d768376563ffab39601050f85e96e2de5aef894b11e475dd85a25"
Jul 17 11:33:29.389805 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:33:29.389771    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45989be5-f5p2q_odh-ai-gateway-infra(61a21f1f-570d-4aef-af72-f8487da613e7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-f5p2q" podUID="61a21f1f-570d-4aef-af72-f8487da613e7"
Jul 17 11:33:30.396149 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:30.396120    2563 scope.go:117] "RemoveContainer" containerID="14d89b10125d768376563ffab39601050f85e96e2de5aef894b11e475dd85a25"
Jul 17 11:33:30.396520 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:33:30.396339    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45989be5-f5p2q_odh-ai-gateway-infra(61a21f1f-570d-4aef-af72-f8487da613e7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-f5p2q" podUID="61a21f1f-570d-4aef-af72-f8487da613e7"
Jul 17 11:33:44.660295 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:44.660264    2563 scope.go:117] "RemoveContainer" containerID="14d89b10125d768376563ffab39601050f85e96e2de5aef894b11e475dd85a25"
Jul 17 11:33:45.475395 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:45.475362    2563 generic.go:356] "Generic (PLEG): container finished" podID="61a21f1f-570d-4aef-af72-f8487da613e7" containerID="b2b10e396d10698ed8bc85b5b085c0ac29e9dbf68d7a1249560b90e858342245" exitCode=7
Jul 17 11:33:45.475592 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:45.475442    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-f5p2q" event={"ID":"61a21f1f-570d-4aef-af72-f8487da613e7","Type":"ContainerDied","Data":"b2b10e396d10698ed8bc85b5b085c0ac29e9dbf68d7a1249560b90e858342245"}
Jul 17 11:33:45.475592 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:45.475515    2563 scope.go:117] "RemoveContainer" containerID="14d89b10125d768376563ffab39601050f85e96e2de5aef894b11e475dd85a25"
Jul 17 11:33:45.475850 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:45.475834    2563 scope.go:117] "RemoveContainer" containerID="b2b10e396d10698ed8bc85b5b085c0ac29e9dbf68d7a1249560b90e858342245"
Jul 17 11:33:45.476033 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:33:45.476004    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45989be5-f5p2q_odh-ai-gateway-infra(61a21f1f-570d-4aef-af72-f8487da613e7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-f5p2q" podUID="61a21f1f-570d-4aef-af72-f8487da613e7"
Jul 17 11:33:46.503615 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:46.503566    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-f5p2q"]
Jul 17 11:33:46.635101 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:46.635059    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-f5p2q"
Jul 17 11:33:46.784212 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:46.784110    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a21f1f-570d-4aef-af72-f8487da613e7-maas-api-service-ca\") pod \"61a21f1f-570d-4aef-af72-f8487da613e7\" (UID: \"61a21f1f-570d-4aef-af72-f8487da613e7\") "
Jul 17 11:33:46.784522 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:46.784496    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61a21f1f-570d-4aef-af72-f8487da613e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "61a21f1f-570d-4aef-af72-f8487da613e7" (UID: "61a21f1f-570d-4aef-af72-f8487da613e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:33:46.884973 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:46.884941    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a21f1f-570d-4aef-af72-f8487da613e7-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:33:47.487503 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:47.487470    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-f5p2q"
Jul 17 11:33:47.487703 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:47.487504    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-f5p2q" event={"ID":"61a21f1f-570d-4aef-af72-f8487da613e7","Type":"ContainerDied","Data":"ca22c05b262383fc90598852e9a383257fb3cf29591297e9af4b08905f1e67ee"}
Jul 17 11:33:47.487703 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:47.487559    2563 scope.go:117] "RemoveContainer" containerID="b2b10e396d10698ed8bc85b5b085c0ac29e9dbf68d7a1249560b90e858342245"
Jul 17 11:33:47.510548 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:47.510515    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-f5p2q"]
Jul 17 11:33:47.518887 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:47.518865    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-f5p2q"]
Jul 17 11:33:47.671427 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:47.671393    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61a21f1f-570d-4aef-af72-f8487da613e7" path="/var/lib/kubelet/pods/61a21f1f-570d-4aef-af72-f8487da613e7/volumes"
Jul 17 11:33:56.192927 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:33:56.192898    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:34:06.969484 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:06.969451    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:34:07.082582 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:07.082545    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:34:12.329758 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:12.329730    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:34:13.418327 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:13.418284    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:34:19.096023 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:19.095982    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:34:19.167296 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:19.167261    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:34:19.374944 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:19.374856    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:34:20.597389 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:20.597352    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:34:20.688844 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:20.688808    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:34:20.762282 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:20.762243    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:34:21.289544 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:21.289508    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:34:21.391355 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:21.391309    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:34:27.038850 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:27.038820    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-vzjtx"]
Jul 17 11:34:27.039272 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:27.039205    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61a21f1f-570d-4aef-af72-f8487da613e7" containerName="revoke-keys"
Jul 17 11:34:27.039272 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:27.039217    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="61a21f1f-570d-4aef-af72-f8487da613e7" containerName="revoke-keys"
Jul 17 11:34:27.039272 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:27.039234    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61a21f1f-570d-4aef-af72-f8487da613e7" containerName="revoke-keys"
Jul 17 11:34:27.039272 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:27.039240    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="61a21f1f-570d-4aef-af72-f8487da613e7" containerName="revoke-keys"
Jul 17 11:34:27.039398 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:27.039294    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="61a21f1f-570d-4aef-af72-f8487da613e7" containerName="revoke-keys"
Jul 17 11:34:27.039398 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:27.039301    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="41224b99-2f2d-45a7-b37c-5d2a59fd2323" containerName="revoke-keys"
Jul 17 11:34:27.039398 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:27.039307    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="61a21f1f-570d-4aef-af72-f8487da613e7" containerName="revoke-keys"
Jul 17 11:34:27.043444 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:27.043426    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-vzjtx"
Jul 17 11:34:27.046293 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:27.046270    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwv2h\""
Jul 17 11:34:27.046512 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:27.046323    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 11:34:27.050339 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:27.050314    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-vzjtx"]
Jul 17 11:34:27.137539 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:27.137504    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1c5059f-1dd4-4505-8138-b2622808f7b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45989be5-vzjtx\" (UID: \"b1c5059f-1dd4-4505-8138-b2622808f7b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-vzjtx"
Jul 17 11:34:27.238433 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:27.238401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1c5059f-1dd4-4505-8138-b2622808f7b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45989be5-vzjtx\" (UID: \"b1c5059f-1dd4-4505-8138-b2622808f7b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-vzjtx"
Jul 17 11:34:27.238980 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:27.238963    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1c5059f-1dd4-4505-8138-b2622808f7b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45989be5-vzjtx\" (UID: \"b1c5059f-1dd4-4505-8138-b2622808f7b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-vzjtx"
Jul 17 11:34:27.354728 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:27.354645    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-vzjtx"
Jul 17 11:34:27.484936 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:27.484912    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-vzjtx"]
Jul 17 11:34:27.486408 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:34:27.486375    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c5059f_1dd4_4505_8138_b2622808f7b3.slice/crio-79f9e9a401f535428506cea70e1e9ddd6ac563a337aebdf8f81417c0fc4b8571 WatchSource:0}: Error finding container 79f9e9a401f535428506cea70e1e9ddd6ac563a337aebdf8f81417c0fc4b8571: Status 404 returned error can't find the container with id 79f9e9a401f535428506cea70e1e9ddd6ac563a337aebdf8f81417c0fc4b8571
Jul 17 11:34:27.685733 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:27.685699    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-vzjtx" event={"ID":"b1c5059f-1dd4-4505-8138-b2622808f7b3","Type":"ContainerStarted","Data":"79f9e9a401f535428506cea70e1e9ddd6ac563a337aebdf8f81417c0fc4b8571"}
Jul 17 11:34:28.692231 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:28.692200    2563 generic.go:356] "Generic (PLEG): container finished" podID="b1c5059f-1dd4-4505-8138-b2622808f7b3" containerID="5311867b8f255e23fa95f9d005f4cae45a8aca518560795b4bf55f85f06f5d54" exitCode=6
Jul 17 11:34:28.692631 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:28.692287    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-vzjtx" event={"ID":"b1c5059f-1dd4-4505-8138-b2622808f7b3","Type":"ContainerDied","Data":"5311867b8f255e23fa95f9d005f4cae45a8aca518560795b4bf55f85f06f5d54"}
Jul 17 11:34:28.692631 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:28.692408    2563 scope.go:117] "RemoveContainer" containerID="5311867b8f255e23fa95f9d005f4cae45a8aca518560795b4bf55f85f06f5d54"
Jul 17 11:34:29.705290 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:29.705253    2563 generic.go:356] "Generic (PLEG): container finished" podID="b1c5059f-1dd4-4505-8138-b2622808f7b3" containerID="b8ce72b2637f74d0a3769bd7169c68a9d6407b4d58c49b8e599b105222134909" exitCode=6
Jul 17 11:34:29.705682 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:29.705320    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-vzjtx" event={"ID":"b1c5059f-1dd4-4505-8138-b2622808f7b3","Type":"ContainerDied","Data":"b8ce72b2637f74d0a3769bd7169c68a9d6407b4d58c49b8e599b105222134909"}
Jul 17 11:34:29.705682 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:29.705391    2563 scope.go:117] "RemoveContainer" containerID="5311867b8f255e23fa95f9d005f4cae45a8aca518560795b4bf55f85f06f5d54"
Jul 17 11:34:29.705682 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:29.705562    2563 scope.go:117] "RemoveContainer" containerID="b8ce72b2637f74d0a3769bd7169c68a9d6407b4d58c49b8e599b105222134909"
Jul 17 11:34:29.705832 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:34:29.705787    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45989be5-vzjtx_odh-ai-gateway-infra(b1c5059f-1dd4-4505-8138-b2622808f7b3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-vzjtx" podUID="b1c5059f-1dd4-4505-8138-b2622808f7b3"
Jul 17 11:34:30.713215 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:30.713186    2563 scope.go:117] "RemoveContainer" containerID="b8ce72b2637f74d0a3769bd7169c68a9d6407b4d58c49b8e599b105222134909"
Jul 17 11:34:30.713649 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:34:30.713378    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45989be5-vzjtx_odh-ai-gateway-infra(b1c5059f-1dd4-4505-8138-b2622808f7b3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-vzjtx" podUID="b1c5059f-1dd4-4505-8138-b2622808f7b3"
Jul 17 11:34:33.638082 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:33.638031    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:34:33.705320 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:33.705282    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:34:37.478441 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:37.478378    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:34:42.660637 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:42.660603    2563 scope.go:117] "RemoveContainer" containerID="b8ce72b2637f74d0a3769bd7169c68a9d6407b4d58c49b8e599b105222134909"
Jul 17 11:34:43.781843 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:43.781812    2563 generic.go:356] "Generic (PLEG): container finished" podID="b1c5059f-1dd4-4505-8138-b2622808f7b3" containerID="24872ca0e5a0ba1b60a5c59e5a304490a7ea292dbf346da84ab6284f919392ae" exitCode=6
Jul 17 11:34:43.782364 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:43.781887    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-vzjtx" event={"ID":"b1c5059f-1dd4-4505-8138-b2622808f7b3","Type":"ContainerDied","Data":"24872ca0e5a0ba1b60a5c59e5a304490a7ea292dbf346da84ab6284f919392ae"}
Jul 17 11:34:43.782364 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:43.781963    2563 scope.go:117] "RemoveContainer" containerID="b8ce72b2637f74d0a3769bd7169c68a9d6407b4d58c49b8e599b105222134909"
Jul 17 11:34:43.782364 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:43.782317    2563 scope.go:117] "RemoveContainer" containerID="24872ca0e5a0ba1b60a5c59e5a304490a7ea292dbf346da84ab6284f919392ae"
Jul 17 11:34:43.782556 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:34:43.782530    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45989be5-vzjtx_odh-ai-gateway-infra(b1c5059f-1dd4-4505-8138-b2622808f7b3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-vzjtx" podUID="b1c5059f-1dd4-4505-8138-b2622808f7b3"
Jul 17 11:34:44.810389 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:44.810354    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-vzjtx"]
Jul 17 11:34:44.944287 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:44.944260    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-vzjtx"
Jul 17 11:34:44.990894 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:44.990868    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1c5059f-1dd4-4505-8138-b2622808f7b3-maas-api-service-ca\") pod \"b1c5059f-1dd4-4505-8138-b2622808f7b3\" (UID: \"b1c5059f-1dd4-4505-8138-b2622808f7b3\") "
Jul 17 11:34:44.991209 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:44.991188    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1c5059f-1dd4-4505-8138-b2622808f7b3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b1c5059f-1dd4-4505-8138-b2622808f7b3" (UID: "b1c5059f-1dd4-4505-8138-b2622808f7b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:34:45.091683 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:45.091607    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1c5059f-1dd4-4505-8138-b2622808f7b3-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:34:45.794984 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:45.794958    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-vzjtx"
Jul 17 11:34:45.795144 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:45.794956    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-vzjtx" event={"ID":"b1c5059f-1dd4-4505-8138-b2622808f7b3","Type":"ContainerDied","Data":"79f9e9a401f535428506cea70e1e9ddd6ac563a337aebdf8f81417c0fc4b8571"}
Jul 17 11:34:45.795144 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:45.795124    2563 scope.go:117] "RemoveContainer" containerID="24872ca0e5a0ba1b60a5c59e5a304490a7ea292dbf346da84ab6284f919392ae"
Jul 17 11:34:45.814521 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:45.814500    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-vzjtx"]
Jul 17 11:34:45.822446 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:45.822424    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-vzjtx"]
Jul 17 11:34:47.665743 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:34:47.665670    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1c5059f-1dd4-4505-8138-b2622808f7b3" path="/var/lib/kubelet/pods/b1c5059f-1dd4-4505-8138-b2622808f7b3/volumes"
Jul 17 11:35:14.977786 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:14.977755    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-92lvl"]
Jul 17 11:35:14.978310 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:14.978128    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1c5059f-1dd4-4505-8138-b2622808f7b3" containerName="revoke-keys"
Jul 17 11:35:14.978310 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:14.978141    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1c5059f-1dd4-4505-8138-b2622808f7b3" containerName="revoke-keys"
Jul 17 11:35:14.978310 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:14.978150    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1c5059f-1dd4-4505-8138-b2622808f7b3" containerName="revoke-keys"
Jul 17 11:35:14.978310 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:14.978156    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1c5059f-1dd4-4505-8138-b2622808f7b3" containerName="revoke-keys"
Jul 17 11:35:14.978310 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:14.978175    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61a21f1f-570d-4aef-af72-f8487da613e7" containerName="revoke-keys"
Jul 17 11:35:14.978310 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:14.978181    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="61a21f1f-570d-4aef-af72-f8487da613e7" containerName="revoke-keys"
Jul 17 11:35:14.978310 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:14.978188    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1c5059f-1dd4-4505-8138-b2622808f7b3" containerName="revoke-keys"
Jul 17 11:35:14.978310 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:14.978193    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1c5059f-1dd4-4505-8138-b2622808f7b3" containerName="revoke-keys"
Jul 17 11:35:14.978310 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:14.978244    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1c5059f-1dd4-4505-8138-b2622808f7b3" containerName="revoke-keys"
Jul 17 11:35:14.978310 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:14.978251    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1c5059f-1dd4-4505-8138-b2622808f7b3" containerName="revoke-keys"
Jul 17 11:35:14.978310 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:14.978258    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="61a21f1f-570d-4aef-af72-f8487da613e7" containerName="revoke-keys"
Jul 17 11:35:14.981091 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:14.981061    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-92lvl"
Jul 17 11:35:14.983547 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:14.983527    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwv2h\""
Jul 17 11:35:14.983672 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:14.983566    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 11:35:14.989436 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:14.989417    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-92lvl"]
Jul 17 11:35:15.052232 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:15.052196    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e97151d8-c91d-40cd-a033-6defe906054a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-57741a73-92lvl\" (UID: \"e97151d8-c91d-40cd-a033-6defe906054a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-92lvl"
Jul 17 11:35:15.153467 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:15.153431    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e97151d8-c91d-40cd-a033-6defe906054a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-57741a73-92lvl\" (UID: \"e97151d8-c91d-40cd-a033-6defe906054a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-92lvl"
Jul 17 11:35:15.154119 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:15.154059    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e97151d8-c91d-40cd-a033-6defe906054a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-57741a73-92lvl\" (UID: \"e97151d8-c91d-40cd-a033-6defe906054a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-92lvl"
Jul 17 11:35:15.292381 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:15.292262    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-92lvl"
Jul 17 11:35:15.428132 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:15.428105    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-92lvl"]
Jul 17 11:35:15.430595 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:35:15.430571    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97151d8_c91d_40cd_a033_6defe906054a.slice/crio-878e86e2cf1b0eb735413cb34de00d2c32772b2fb1d0202a37f76d81fa03925b WatchSource:0}: Error finding container 878e86e2cf1b0eb735413cb34de00d2c32772b2fb1d0202a37f76d81fa03925b: Status 404 returned error can't find the container with id 878e86e2cf1b0eb735413cb34de00d2c32772b2fb1d0202a37f76d81fa03925b
Jul 17 11:35:15.953532 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:15.953449    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-92lvl" event={"ID":"e97151d8-c91d-40cd-a033-6defe906054a","Type":"ContainerStarted","Data":"765a76a2e26618cb36289e41be1627faa85a04bdc75d91c0d3562f032eb3c840"}
Jul 17 11:35:15.953532 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:15.953514    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-92lvl" event={"ID":"e97151d8-c91d-40cd-a033-6defe906054a","Type":"ContainerStarted","Data":"878e86e2cf1b0eb735413cb34de00d2c32772b2fb1d0202a37f76d81fa03925b"}
Jul 17 11:35:15.969876 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:15.969826    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-92lvl" podStartSLOduration=1.969812991 podStartE2EDuration="1.969812991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:35:15.969455347 +0000 UTC m=+2270.898994989" watchObservedRunningTime="2026-07-17 11:35:15.969812991 +0000 UTC m=+2270.899352630"
Jul 17 11:35:16.960514 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:16.960480    2563 generic.go:356] "Generic (PLEG): container finished" podID="e97151d8-c91d-40cd-a033-6defe906054a" containerID="765a76a2e26618cb36289e41be1627faa85a04bdc75d91c0d3562f032eb3c840" exitCode=7
Jul 17 11:35:16.960888 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:16.960523    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-92lvl" event={"ID":"e97151d8-c91d-40cd-a033-6defe906054a","Type":"ContainerDied","Data":"765a76a2e26618cb36289e41be1627faa85a04bdc75d91c0d3562f032eb3c840"}
Jul 17 11:35:16.960888 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:16.960699    2563 scope.go:117] "RemoveContainer" containerID="765a76a2e26618cb36289e41be1627faa85a04bdc75d91c0d3562f032eb3c840"
Jul 17 11:35:17.487895 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:17.487860    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:35:17.967556 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:17.967523    2563 generic.go:356] "Generic (PLEG): container finished" podID="e97151d8-c91d-40cd-a033-6defe906054a" containerID="e2ed21c72213abf5e81ac5d183d4dd43179e48fac94b332940ffe77e9018aaec" exitCode=7
Jul 17 11:35:17.967944 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:17.967610    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-92lvl" event={"ID":"e97151d8-c91d-40cd-a033-6defe906054a","Type":"ContainerDied","Data":"e2ed21c72213abf5e81ac5d183d4dd43179e48fac94b332940ffe77e9018aaec"}
Jul 17 11:35:17.967944 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:17.967673    2563 scope.go:117] "RemoveContainer" containerID="765a76a2e26618cb36289e41be1627faa85a04bdc75d91c0d3562f032eb3c840"
Jul 17 11:35:17.967944 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:17.967862    2563 scope.go:117] "RemoveContainer" containerID="e2ed21c72213abf5e81ac5d183d4dd43179e48fac94b332940ffe77e9018aaec"
Jul 17 11:35:17.968139 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:35:17.968107    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-57741a73-92lvl_odh-ai-gateway-infra(e97151d8-c91d-40cd-a033-6defe906054a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-92lvl" podUID="e97151d8-c91d-40cd-a033-6defe906054a"
Jul 17 11:35:18.974298 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:18.974273    2563 scope.go:117] "RemoveContainer" containerID="e2ed21c72213abf5e81ac5d183d4dd43179e48fac94b332940ffe77e9018aaec"
Jul 17 11:35:18.974684 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:35:18.974452    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-57741a73-92lvl_odh-ai-gateway-infra(e97151d8-c91d-40cd-a033-6defe906054a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-92lvl" podUID="e97151d8-c91d-40cd-a033-6defe906054a"
Jul 17 11:35:32.659946 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:32.659911    2563 scope.go:117] "RemoveContainer" containerID="e2ed21c72213abf5e81ac5d183d4dd43179e48fac94b332940ffe77e9018aaec"
Jul 17 11:35:33.051797 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:33.051764    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-92lvl" event={"ID":"e97151d8-c91d-40cd-a033-6defe906054a","Type":"ContainerStarted","Data":"ae134f299ed82420b5598bc946e9687d44dab401c62fb1b93d79493953d9adb4"}
Jul 17 11:35:34.059246 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:34.059207    2563 generic.go:356] "Generic (PLEG): container finished" podID="e97151d8-c91d-40cd-a033-6defe906054a" containerID="ae134f299ed82420b5598bc946e9687d44dab401c62fb1b93d79493953d9adb4" exitCode=7
Jul 17 11:35:34.059246 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:34.059251    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-92lvl" event={"ID":"e97151d8-c91d-40cd-a033-6defe906054a","Type":"ContainerDied","Data":"ae134f299ed82420b5598bc946e9687d44dab401c62fb1b93d79493953d9adb4"}
Jul 17 11:35:34.059696 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:34.059300    2563 scope.go:117] "RemoveContainer" containerID="e2ed21c72213abf5e81ac5d183d4dd43179e48fac94b332940ffe77e9018aaec"
Jul 17 11:35:34.059696 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:34.059590    2563 scope.go:117] "RemoveContainer" containerID="ae134f299ed82420b5598bc946e9687d44dab401c62fb1b93d79493953d9adb4"
Jul 17 11:35:34.059837 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:35:34.059817    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-57741a73-92lvl_odh-ai-gateway-infra(e97151d8-c91d-40cd-a033-6defe906054a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-92lvl" podUID="e97151d8-c91d-40cd-a033-6defe906054a"
Jul 17 11:35:34.081691 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:34.081662    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-92lvl"]
Jul 17 11:35:35.198465 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:35.198438    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-92lvl"
Jul 17 11:35:35.337094 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:35.336989    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e97151d8-c91d-40cd-a033-6defe906054a-maas-api-service-ca\") pod \"e97151d8-c91d-40cd-a033-6defe906054a\" (UID: \"e97151d8-c91d-40cd-a033-6defe906054a\") "
Jul 17 11:35:35.337403 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:35.337381    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e97151d8-c91d-40cd-a033-6defe906054a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e97151d8-c91d-40cd-a033-6defe906054a" (UID: "e97151d8-c91d-40cd-a033-6defe906054a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:35:35.438687 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:35.438655    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e97151d8-c91d-40cd-a033-6defe906054a-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:35:36.073932 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:36.073824    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-92lvl" event={"ID":"e97151d8-c91d-40cd-a033-6defe906054a","Type":"ContainerDied","Data":"878e86e2cf1b0eb735413cb34de00d2c32772b2fb1d0202a37f76d81fa03925b"}
Jul 17 11:35:36.074145 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:36.073856    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-92lvl"
Jul 17 11:35:36.074145 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:36.074132    2563 scope.go:117] "RemoveContainer" containerID="ae134f299ed82420b5598bc946e9687d44dab401c62fb1b93d79493953d9adb4"
Jul 17 11:35:36.096447 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:36.096419    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-92lvl"]
Jul 17 11:35:36.100881 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:36.100859    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-92lvl"]
Jul 17 11:35:37.665914 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:37.665881    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e97151d8-c91d-40cd-a033-6defe906054a" path="/var/lib/kubelet/pods/e97151d8-c91d-40cd-a033-6defe906054a/volumes"
Jul 17 11:35:58.248475 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:58.248440    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t"]
Jul 17 11:35:58.248965 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:58.248755    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e97151d8-c91d-40cd-a033-6defe906054a" containerName="revoke-keys"
Jul 17 11:35:58.248965 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:58.248766    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e97151d8-c91d-40cd-a033-6defe906054a" containerName="revoke-keys"
Jul 17 11:35:58.248965 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:58.248775    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e97151d8-c91d-40cd-a033-6defe906054a" containerName="revoke-keys"
Jul 17 11:35:58.248965 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:58.248781    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e97151d8-c91d-40cd-a033-6defe906054a" containerName="revoke-keys"
Jul 17 11:35:58.248965 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:58.248790    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e97151d8-c91d-40cd-a033-6defe906054a" containerName="revoke-keys"
Jul 17 11:35:58.248965 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:58.248795    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e97151d8-c91d-40cd-a033-6defe906054a" containerName="revoke-keys"
Jul 17 11:35:58.248965 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:58.248851    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e97151d8-c91d-40cd-a033-6defe906054a" containerName="revoke-keys"
Jul 17 11:35:58.248965 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:58.248859    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e97151d8-c91d-40cd-a033-6defe906054a" containerName="revoke-keys"
Jul 17 11:35:58.248965 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:58.248865    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1c5059f-1dd4-4505-8138-b2622808f7b3" containerName="revoke-keys"
Jul 17 11:35:58.253248 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:58.253232    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t"
Jul 17 11:35:58.255728 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:58.255701    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 11:35:58.255854 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:58.255748    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwv2h\""
Jul 17 11:35:58.262468 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:58.262446    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t"]
Jul 17 11:35:58.334761 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:58.334720    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c906c0c2-cd53-4999-9b5a-22c13f9e89af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t\" (UID: \"c906c0c2-cd53-4999-9b5a-22c13f9e89af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t"
Jul 17 11:35:58.435784 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:58.435752    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c906c0c2-cd53-4999-9b5a-22c13f9e89af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t\" (UID: \"c906c0c2-cd53-4999-9b5a-22c13f9e89af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t"
Jul 17 11:35:58.436380 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:58.436361    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c906c0c2-cd53-4999-9b5a-22c13f9e89af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t\" (UID: \"c906c0c2-cd53-4999-9b5a-22c13f9e89af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t"
Jul 17 11:35:58.564423 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:58.564351    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t"
Jul 17 11:35:58.691766 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:58.691743    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t"]
Jul 17 11:35:58.693623 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:35:58.693589    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc906c0c2_cd53_4999_9b5a_22c13f9e89af.slice/crio-4c2d77886069852966f4471d39048d96c01c4dc2ec3ced9f0c81cacdf94ff368 WatchSource:0}: Error finding container 4c2d77886069852966f4471d39048d96c01c4dc2ec3ced9f0c81cacdf94ff368: Status 404 returned error can't find the container with id 4c2d77886069852966f4471d39048d96c01c4dc2ec3ced9f0c81cacdf94ff368
Jul 17 11:35:59.188609 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:59.188578    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t" event={"ID":"c906c0c2-cd53-4999-9b5a-22c13f9e89af","Type":"ContainerStarted","Data":"e0bd9c6528b982d7efa1f9852635b2374fb7acd9f96507e6ad2fede98fd0e84d"}
Jul 17 11:35:59.188800 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:59.188637    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t" event={"ID":"c906c0c2-cd53-4999-9b5a-22c13f9e89af","Type":"ContainerStarted","Data":"4c2d77886069852966f4471d39048d96c01c4dc2ec3ced9f0c81cacdf94ff368"}
Jul 17 11:35:59.205768 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:35:59.205715    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t" podStartSLOduration=1.205699159 podStartE2EDuration="1.205699159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:35:59.205127353 +0000 UTC m=+2314.134666995" watchObservedRunningTime="2026-07-17 11:35:59.205699159 +0000 UTC m=+2314.135238801"
Jul 17 11:36:00.194713 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:00.194678    2563 generic.go:356] "Generic (PLEG): container finished" podID="c906c0c2-cd53-4999-9b5a-22c13f9e89af" containerID="e0bd9c6528b982d7efa1f9852635b2374fb7acd9f96507e6ad2fede98fd0e84d" exitCode=6
Jul 17 11:36:00.195176 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:00.194741    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t" event={"ID":"c906c0c2-cd53-4999-9b5a-22c13f9e89af","Type":"ContainerDied","Data":"e0bd9c6528b982d7efa1f9852635b2374fb7acd9f96507e6ad2fede98fd0e84d"}
Jul 17 11:36:00.195176 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:00.194946    2563 scope.go:117] "RemoveContainer" containerID="e0bd9c6528b982d7efa1f9852635b2374fb7acd9f96507e6ad2fede98fd0e84d"
Jul 17 11:36:01.201517 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:01.201483    2563 generic.go:356] "Generic (PLEG): container finished" podID="c906c0c2-cd53-4999-9b5a-22c13f9e89af" containerID="ecb967901a18294e1f39c62a736a41b06163159cc83481a297c3b332cd2b72a9" exitCode=6
Jul 17 11:36:01.201895 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:01.201561    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t" event={"ID":"c906c0c2-cd53-4999-9b5a-22c13f9e89af","Type":"ContainerDied","Data":"ecb967901a18294e1f39c62a736a41b06163159cc83481a297c3b332cd2b72a9"}
Jul 17 11:36:01.201895 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:01.201615    2563 scope.go:117] "RemoveContainer" containerID="e0bd9c6528b982d7efa1f9852635b2374fb7acd9f96507e6ad2fede98fd0e84d"
Jul 17 11:36:01.201895 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:01.201820    2563 scope.go:117] "RemoveContainer" containerID="ecb967901a18294e1f39c62a736a41b06163159cc83481a297c3b332cd2b72a9"
Jul 17 11:36:01.202117 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:36:01.202095    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t_odh-ai-gateway-infra(c906c0c2-cd53-4999-9b5a-22c13f9e89af)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t" podUID="c906c0c2-cd53-4999-9b5a-22c13f9e89af"
Jul 17 11:36:02.208116 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:02.208062    2563 scope.go:117] "RemoveContainer" containerID="ecb967901a18294e1f39c62a736a41b06163159cc83481a297c3b332cd2b72a9"
Jul 17 11:36:02.208477 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:36:02.208264    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t_odh-ai-gateway-infra(c906c0c2-cd53-4999-9b5a-22c13f9e89af)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t" podUID="c906c0c2-cd53-4999-9b5a-22c13f9e89af"
Jul 17 11:36:14.660338 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:14.660303    2563 scope.go:117] "RemoveContainer" containerID="ecb967901a18294e1f39c62a736a41b06163159cc83481a297c3b332cd2b72a9"
Jul 17 11:36:15.272252 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:15.272226    2563 generic.go:356] "Generic (PLEG): container finished" podID="c906c0c2-cd53-4999-9b5a-22c13f9e89af" containerID="306d3aa134748dc3efe5bb1ec4a7526d16ef6b951fcf3c8e01a29ca49cbf487e" exitCode=6
Jul 17 11:36:15.272366 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:15.272300    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t" event={"ID":"c906c0c2-cd53-4999-9b5a-22c13f9e89af","Type":"ContainerDied","Data":"306d3aa134748dc3efe5bb1ec4a7526d16ef6b951fcf3c8e01a29ca49cbf487e"}
Jul 17 11:36:15.272366 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:15.272359    2563 scope.go:117] "RemoveContainer" containerID="ecb967901a18294e1f39c62a736a41b06163159cc83481a297c3b332cd2b72a9"
Jul 17 11:36:15.272659 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:15.272643    2563 scope.go:117] "RemoveContainer" containerID="306d3aa134748dc3efe5bb1ec4a7526d16ef6b951fcf3c8e01a29ca49cbf487e"
Jul 17 11:36:15.272870 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:36:15.272850    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t_odh-ai-gateway-infra(c906c0c2-cd53-4999-9b5a-22c13f9e89af)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t" podUID="c906c0c2-cd53-4999-9b5a-22c13f9e89af"
Jul 17 11:36:16.300656 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:16.300626    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t"]
Jul 17 11:36:16.448047 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:16.448025    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t"
Jul 17 11:36:16.594588 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:16.594500    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c906c0c2-cd53-4999-9b5a-22c13f9e89af-maas-api-service-ca\") pod \"c906c0c2-cd53-4999-9b5a-22c13f9e89af\" (UID: \"c906c0c2-cd53-4999-9b5a-22c13f9e89af\") "
Jul 17 11:36:16.594869 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:16.594845    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c906c0c2-cd53-4999-9b5a-22c13f9e89af-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c906c0c2-cd53-4999-9b5a-22c13f9e89af" (UID: "c906c0c2-cd53-4999-9b5a-22c13f9e89af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:36:16.695935 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:16.695898    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c906c0c2-cd53-4999-9b5a-22c13f9e89af-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:36:17.284260 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:17.284229    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t"
Jul 17 11:36:17.284260 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:17.284247    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t" event={"ID":"c906c0c2-cd53-4999-9b5a-22c13f9e89af","Type":"ContainerDied","Data":"4c2d77886069852966f4471d39048d96c01c4dc2ec3ced9f0c81cacdf94ff368"}
Jul 17 11:36:17.284508 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:17.284301    2563 scope.go:117] "RemoveContainer" containerID="306d3aa134748dc3efe5bb1ec4a7526d16ef6b951fcf3c8e01a29ca49cbf487e"
Jul 17 11:36:17.306753 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:17.306728    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t"]
Jul 17 11:36:17.311891 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:17.311869    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-p4d8t"]
Jul 17 11:36:17.665112 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:17.665056    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c906c0c2-cd53-4999-9b5a-22c13f9e89af" path="/var/lib/kubelet/pods/c906c0c2-cd53-4999-9b5a-22c13f9e89af/volumes"
Jul 17 11:36:18.310644 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:18.310606    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-m5nct"]
Jul 17 11:36:18.311156 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:18.311106    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c906c0c2-cd53-4999-9b5a-22c13f9e89af" containerName="revoke-keys"
Jul 17 11:36:18.311156 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:18.311125    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c906c0c2-cd53-4999-9b5a-22c13f9e89af" containerName="revoke-keys"
Jul 17 11:36:18.311156 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:18.311143    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c906c0c2-cd53-4999-9b5a-22c13f9e89af" containerName="revoke-keys"
Jul 17 11:36:18.311156 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:18.311154    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c906c0c2-cd53-4999-9b5a-22c13f9e89af" containerName="revoke-keys"
Jul 17 11:36:18.311373 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:18.311283    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c906c0c2-cd53-4999-9b5a-22c13f9e89af" containerName="revoke-keys"
Jul 17 11:36:18.311373 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:18.311300    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e97151d8-c91d-40cd-a033-6defe906054a" containerName="revoke-keys"
Jul 17 11:36:18.311373 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:18.311310    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c906c0c2-cd53-4999-9b5a-22c13f9e89af" containerName="revoke-keys"
Jul 17 11:36:18.315412 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:18.315392    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-m5nct"
Jul 17 11:36:18.319837 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:18.319814    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwv2h\""
Jul 17 11:36:18.325239 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:18.325212    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 11:36:18.326965 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:18.326943    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-m5nct"]
Jul 17 11:36:18.412589 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:18.412551    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4934625-1520-4572-ac15-08d48fa843b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-829f05b9-m5nct\" (UID: \"e4934625-1520-4572-ac15-08d48fa843b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-m5nct"
Jul 17 11:36:18.513777 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:18.513726    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4934625-1520-4572-ac15-08d48fa843b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-829f05b9-m5nct\" (UID: \"e4934625-1520-4572-ac15-08d48fa843b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-m5nct"
Jul 17 11:36:18.514482 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:18.514463    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4934625-1520-4572-ac15-08d48fa843b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-829f05b9-m5nct\" (UID: \"e4934625-1520-4572-ac15-08d48fa843b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-m5nct"
Jul 17 11:36:18.625938 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:18.625848    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-m5nct"
Jul 17 11:36:18.758544 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:18.758520    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-m5nct"]
Jul 17 11:36:18.760401 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:36:18.760374    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4934625_1520_4572_ac15_08d48fa843b4.slice/crio-750a8d8b9c1900b8279fb7b425404a0b3ceeaa3c8e802bd79ebf57155fae30d1 WatchSource:0}: Error finding container 750a8d8b9c1900b8279fb7b425404a0b3ceeaa3c8e802bd79ebf57155fae30d1: Status 404 returned error can't find the container with id 750a8d8b9c1900b8279fb7b425404a0b3ceeaa3c8e802bd79ebf57155fae30d1
Jul 17 11:36:19.297180 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:19.297083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-m5nct" event={"ID":"e4934625-1520-4572-ac15-08d48fa843b4","Type":"ContainerStarted","Data":"2995c448bbd2079fca2ab602d21abf1790a842060cc6ca3dbc6e22f5f33c1a74"}
Jul 17 11:36:19.297180 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:19.297145    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-m5nct" event={"ID":"e4934625-1520-4572-ac15-08d48fa843b4","Type":"ContainerStarted","Data":"750a8d8b9c1900b8279fb7b425404a0b3ceeaa3c8e802bd79ebf57155fae30d1"}
Jul 17 11:36:19.315847 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:19.315794    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-m5nct" podStartSLOduration=1.315779555 podStartE2EDuration="1.315779555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:36:19.313796589 +0000 UTC m=+2334.243336229" watchObservedRunningTime="2026-07-17 11:36:19.315779555 +0000 UTC m=+2334.245319195"
Jul 17 11:36:20.303977 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:20.303944    2563 generic.go:356] "Generic (PLEG): container finished" podID="e4934625-1520-4572-ac15-08d48fa843b4" containerID="2995c448bbd2079fca2ab602d21abf1790a842060cc6ca3dbc6e22f5f33c1a74" exitCode=6
Jul 17 11:36:20.304220 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:20.304011    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-m5nct" event={"ID":"e4934625-1520-4572-ac15-08d48fa843b4","Type":"ContainerDied","Data":"2995c448bbd2079fca2ab602d21abf1790a842060cc6ca3dbc6e22f5f33c1a74"}
Jul 17 11:36:20.304220 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:20.304212    2563 scope.go:117] "RemoveContainer" containerID="2995c448bbd2079fca2ab602d21abf1790a842060cc6ca3dbc6e22f5f33c1a74"
Jul 17 11:36:21.311048 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:21.310950    2563 generic.go:356] "Generic (PLEG): container finished" podID="e4934625-1520-4572-ac15-08d48fa843b4" containerID="fd1aa3e1706a5cbf621b7686d6fe779ef7dc9dd69a570aaa3905fa4a39a113d3" exitCode=6
Jul 17 11:36:21.311479 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:21.311042    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-m5nct" event={"ID":"e4934625-1520-4572-ac15-08d48fa843b4","Type":"ContainerDied","Data":"fd1aa3e1706a5cbf621b7686d6fe779ef7dc9dd69a570aaa3905fa4a39a113d3"}
Jul 17 11:36:21.311479 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:21.311125    2563 scope.go:117] "RemoveContainer" containerID="2995c448bbd2079fca2ab602d21abf1790a842060cc6ca3dbc6e22f5f33c1a74"
Jul 17 11:36:21.311479 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:21.311281    2563 scope.go:117] "RemoveContainer" containerID="fd1aa3e1706a5cbf621b7686d6fe779ef7dc9dd69a570aaa3905fa4a39a113d3"
Jul 17 11:36:21.311634 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:36:21.311482    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-829f05b9-m5nct_odh-ai-gateway-infra(e4934625-1520-4572-ac15-08d48fa843b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-m5nct" podUID="e4934625-1520-4572-ac15-08d48fa843b4"
Jul 17 11:36:22.317878 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:22.317847    2563 scope.go:117] "RemoveContainer" containerID="fd1aa3e1706a5cbf621b7686d6fe779ef7dc9dd69a570aaa3905fa4a39a113d3"
Jul 17 11:36:22.318288 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:36:22.318055    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-829f05b9-m5nct_odh-ai-gateway-infra(e4934625-1520-4572-ac15-08d48fa843b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-m5nct" podUID="e4934625-1520-4572-ac15-08d48fa843b4"
Jul 17 11:36:36.659633 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:36.659606    2563 scope.go:117] "RemoveContainer" containerID="fd1aa3e1706a5cbf621b7686d6fe779ef7dc9dd69a570aaa3905fa4a39a113d3"
Jul 17 11:36:37.399024 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:37.398994    2563 generic.go:356] "Generic (PLEG): container finished" podID="e4934625-1520-4572-ac15-08d48fa843b4" containerID="c648c862927ad42c66d8a136e1b4b6f1eb6f53f3b1ea9237d9c03b5035a0b00c" exitCode=6
Jul 17 11:36:37.399231 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:37.399064    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-m5nct" event={"ID":"e4934625-1520-4572-ac15-08d48fa843b4","Type":"ContainerDied","Data":"c648c862927ad42c66d8a136e1b4b6f1eb6f53f3b1ea9237d9c03b5035a0b00c"}
Jul 17 11:36:37.399231 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:37.399145    2563 scope.go:117] "RemoveContainer" containerID="fd1aa3e1706a5cbf621b7686d6fe779ef7dc9dd69a570aaa3905fa4a39a113d3"
Jul 17 11:36:37.399490 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:37.399463    2563 scope.go:117] "RemoveContainer" containerID="c648c862927ad42c66d8a136e1b4b6f1eb6f53f3b1ea9237d9c03b5035a0b00c"
Jul 17 11:36:37.399701 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:36:37.399679    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-829f05b9-m5nct_odh-ai-gateway-infra(e4934625-1520-4572-ac15-08d48fa843b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-m5nct" podUID="e4934625-1520-4572-ac15-08d48fa843b4"
Jul 17 11:36:38.426349 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:38.426315    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-m5nct"]
Jul 17 11:36:38.552557 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:38.552531    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-m5nct"
Jul 17 11:36:38.575455 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:38.575426    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4934625-1520-4572-ac15-08d48fa843b4-maas-api-service-ca\") pod \"e4934625-1520-4572-ac15-08d48fa843b4\" (UID: \"e4934625-1520-4572-ac15-08d48fa843b4\") "
Jul 17 11:36:38.575847 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:38.575824    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4934625-1520-4572-ac15-08d48fa843b4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e4934625-1520-4572-ac15-08d48fa843b4" (UID: "e4934625-1520-4572-ac15-08d48fa843b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:36:38.676047 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:38.676011    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4934625-1520-4572-ac15-08d48fa843b4-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:36:39.411349 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:39.411314    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-m5nct" event={"ID":"e4934625-1520-4572-ac15-08d48fa843b4","Type":"ContainerDied","Data":"750a8d8b9c1900b8279fb7b425404a0b3ceeaa3c8e802bd79ebf57155fae30d1"}
Jul 17 11:36:39.411541 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:39.411379    2563 scope.go:117] "RemoveContainer" containerID="c648c862927ad42c66d8a136e1b4b6f1eb6f53f3b1ea9237d9c03b5035a0b00c"
Jul 17 11:36:39.411541 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:39.411325    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-m5nct"
Jul 17 11:36:39.435202 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:39.435172    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-m5nct"]
Jul 17 11:36:39.443118 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:39.443095    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-m5nct"]
Jul 17 11:36:39.665273 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:39.665197    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4934625-1520-4572-ac15-08d48fa843b4" path="/var/lib/kubelet/pods/e4934625-1520-4572-ac15-08d48fa843b4/volumes"
Jul 17 11:36:48.321158 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:48.321123    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-6t2pz"]
Jul 17 11:36:48.321534 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:48.321431    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4934625-1520-4572-ac15-08d48fa843b4" containerName="revoke-keys"
Jul 17 11:36:48.321534 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:48.321441    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4934625-1520-4572-ac15-08d48fa843b4" containerName="revoke-keys"
Jul 17 11:36:48.321534 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:48.321451    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4934625-1520-4572-ac15-08d48fa843b4" containerName="revoke-keys"
Jul 17 11:36:48.321534 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:48.321457    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4934625-1520-4572-ac15-08d48fa843b4" containerName="revoke-keys"
Jul 17 11:36:48.321534 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:48.321473    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4934625-1520-4572-ac15-08d48fa843b4" containerName="revoke-keys"
Jul 17 11:36:48.321534 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:48.321478    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4934625-1520-4572-ac15-08d48fa843b4" containerName="revoke-keys"
Jul 17 11:36:48.321534 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:48.321485    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c906c0c2-cd53-4999-9b5a-22c13f9e89af" containerName="revoke-keys"
Jul 17 11:36:48.321534 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:48.321490    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c906c0c2-cd53-4999-9b5a-22c13f9e89af" containerName="revoke-keys"
Jul 17 11:36:48.321777 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:48.321540    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4934625-1520-4572-ac15-08d48fa843b4" containerName="revoke-keys"
Jul 17 11:36:48.321777 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:48.321546    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c906c0c2-cd53-4999-9b5a-22c13f9e89af" containerName="revoke-keys"
Jul 17 11:36:48.321777 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:48.321553    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4934625-1520-4572-ac15-08d48fa843b4" containerName="revoke-keys"
Jul 17 11:36:48.321777 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:48.321559    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4934625-1520-4572-ac15-08d48fa843b4" containerName="revoke-keys"
Jul 17 11:36:48.325810 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:48.325792    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-6t2pz"
Jul 17 11:36:48.331227 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:48.331201    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 11:36:48.331227 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:48.331217    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwv2h\""
Jul 17 11:36:48.353586 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:48.353559    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-6t2pz"]
Jul 17 11:36:48.358232 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:48.358204    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a92901a5-77cd-4d34-b6f8-85b43bcb99c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-40b693f1-6t2pz\" (UID: \"a92901a5-77cd-4d34-b6f8-85b43bcb99c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-6t2pz"
Jul 17 11:36:48.459442 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:48.459414    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a92901a5-77cd-4d34-b6f8-85b43bcb99c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-40b693f1-6t2pz\" (UID: \"a92901a5-77cd-4d34-b6f8-85b43bcb99c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-6t2pz"
Jul 17 11:36:48.460037 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:48.460019    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a92901a5-77cd-4d34-b6f8-85b43bcb99c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-40b693f1-6t2pz\" (UID: \"a92901a5-77cd-4d34-b6f8-85b43bcb99c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-6t2pz"
Jul 17 11:36:48.635304 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:48.635278    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-6t2pz"
Jul 17 11:36:48.780393 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:48.780370    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-6t2pz"]
Jul 17 11:36:48.782496 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:36:48.782452    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda92901a5_77cd_4d34_b6f8_85b43bcb99c1.slice/crio-e52f1418b9a33bbf9c08fc625a2ff6834a1c6f8765845da06dd7aae70e09af35 WatchSource:0}: Error finding container e52f1418b9a33bbf9c08fc625a2ff6834a1c6f8765845da06dd7aae70e09af35: Status 404 returned error can't find the container with id e52f1418b9a33bbf9c08fc625a2ff6834a1c6f8765845da06dd7aae70e09af35
Jul 17 11:36:49.462136 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:49.462108    2563 generic.go:356] "Generic (PLEG): container finished" podID="a92901a5-77cd-4d34-b6f8-85b43bcb99c1" containerID="1727ac7ddb9be8c099199aa06a886a3808c3849f566c358a460950c07b32e17c" exitCode=6
Jul 17 11:36:49.462540 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:49.462203    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-6t2pz" event={"ID":"a92901a5-77cd-4d34-b6f8-85b43bcb99c1","Type":"ContainerDied","Data":"1727ac7ddb9be8c099199aa06a886a3808c3849f566c358a460950c07b32e17c"}
Jul 17 11:36:49.462540 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:49.462269    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-6t2pz" event={"ID":"a92901a5-77cd-4d34-b6f8-85b43bcb99c1","Type":"ContainerStarted","Data":"e52f1418b9a33bbf9c08fc625a2ff6834a1c6f8765845da06dd7aae70e09af35"}
Jul 17 11:36:49.462540 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:49.462372    2563 scope.go:117] "RemoveContainer" containerID="1727ac7ddb9be8c099199aa06a886a3808c3849f566c358a460950c07b32e17c"
Jul 17 11:36:50.468611 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:50.468512    2563 generic.go:356] "Generic (PLEG): container finished" podID="a92901a5-77cd-4d34-b6f8-85b43bcb99c1" containerID="98f24cfa86892aaaf81e78b0d81e19c397abb206387d3dad2720eb6c13ba0046" exitCode=6
Jul 17 11:36:50.468984 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:50.468601    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-6t2pz" event={"ID":"a92901a5-77cd-4d34-b6f8-85b43bcb99c1","Type":"ContainerDied","Data":"98f24cfa86892aaaf81e78b0d81e19c397abb206387d3dad2720eb6c13ba0046"}
Jul 17 11:36:50.468984 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:50.468657    2563 scope.go:117] "RemoveContainer" containerID="1727ac7ddb9be8c099199aa06a886a3808c3849f566c358a460950c07b32e17c"
Jul 17 11:36:50.468984 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:50.468796    2563 scope.go:117] "RemoveContainer" containerID="98f24cfa86892aaaf81e78b0d81e19c397abb206387d3dad2720eb6c13ba0046"
Jul 17 11:36:50.469126 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:36:50.469026    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-40b693f1-6t2pz_odh-ai-gateway-infra(a92901a5-77cd-4d34-b6f8-85b43bcb99c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-6t2pz" podUID="a92901a5-77cd-4d34-b6f8-85b43bcb99c1"
Jul 17 11:36:51.475194 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:51.475165    2563 scope.go:117] "RemoveContainer" containerID="98f24cfa86892aaaf81e78b0d81e19c397abb206387d3dad2720eb6c13ba0046"
Jul 17 11:36:51.475594 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:36:51.475358    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-40b693f1-6t2pz_odh-ai-gateway-infra(a92901a5-77cd-4d34-b6f8-85b43bcb99c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-6t2pz" podUID="a92901a5-77cd-4d34-b6f8-85b43bcb99c1"
Jul 17 11:36:51.595408 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:51.595368    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:36:53.242433 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:53.242402    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:36:55.689011 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:36:55.688980    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:37:02.660197 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:02.660157    2563 scope.go:117] "RemoveContainer" containerID="98f24cfa86892aaaf81e78b0d81e19c397abb206387d3dad2720eb6c13ba0046"
Jul 17 11:37:03.533189 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:03.533105    2563 generic.go:356] "Generic (PLEG): container finished" podID="a92901a5-77cd-4d34-b6f8-85b43bcb99c1" containerID="f6d1deeb332e7b5c508f047ce52b834a14e07b24f0a2aa336423a0346a1fd65e" exitCode=6
Jul 17 11:37:03.533330 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:03.533180    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-6t2pz" event={"ID":"a92901a5-77cd-4d34-b6f8-85b43bcb99c1","Type":"ContainerDied","Data":"f6d1deeb332e7b5c508f047ce52b834a14e07b24f0a2aa336423a0346a1fd65e"}
Jul 17 11:37:03.533330 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:03.533239    2563 scope.go:117] "RemoveContainer" containerID="98f24cfa86892aaaf81e78b0d81e19c397abb206387d3dad2720eb6c13ba0046"
Jul 17 11:37:03.533542 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:03.533512    2563 scope.go:117] "RemoveContainer" containerID="f6d1deeb332e7b5c508f047ce52b834a14e07b24f0a2aa336423a0346a1fd65e"
Jul 17 11:37:03.533773 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:37:03.533748    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-40b693f1-6t2pz_odh-ai-gateway-infra(a92901a5-77cd-4d34-b6f8-85b43bcb99c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-6t2pz" podUID="a92901a5-77cd-4d34-b6f8-85b43bcb99c1"
Jul 17 11:37:04.574339 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:04.574303    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-6t2pz"]
Jul 17 11:37:04.703629 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:04.703606    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-6t2pz"
Jul 17 11:37:04.810352 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:04.810323    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a92901a5-77cd-4d34-b6f8-85b43bcb99c1-maas-api-service-ca\") pod \"a92901a5-77cd-4d34-b6f8-85b43bcb99c1\" (UID: \"a92901a5-77cd-4d34-b6f8-85b43bcb99c1\") "
Jul 17 11:37:04.810678 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:04.810652    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a92901a5-77cd-4d34-b6f8-85b43bcb99c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a92901a5-77cd-4d34-b6f8-85b43bcb99c1" (UID: "a92901a5-77cd-4d34-b6f8-85b43bcb99c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:37:04.911601 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:04.911572    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a92901a5-77cd-4d34-b6f8-85b43bcb99c1-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:37:05.098742 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:05.098707    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-r7fk7"]
Jul 17 11:37:05.099053 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:05.099041    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a92901a5-77cd-4d34-b6f8-85b43bcb99c1" containerName="revoke-keys"
Jul 17 11:37:05.099118 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:05.099054    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a92901a5-77cd-4d34-b6f8-85b43bcb99c1" containerName="revoke-keys"
Jul 17 11:37:05.099118 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:05.099088    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a92901a5-77cd-4d34-b6f8-85b43bcb99c1" containerName="revoke-keys"
Jul 17 11:37:05.099118 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:05.099094    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a92901a5-77cd-4d34-b6f8-85b43bcb99c1" containerName="revoke-keys"
Jul 17 11:37:05.099118 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:05.099102    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a92901a5-77cd-4d34-b6f8-85b43bcb99c1" containerName="revoke-keys"
Jul 17 11:37:05.099118 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:05.099108    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a92901a5-77cd-4d34-b6f8-85b43bcb99c1" containerName="revoke-keys"
Jul 17 11:37:05.099280 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:05.099172    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a92901a5-77cd-4d34-b6f8-85b43bcb99c1" containerName="revoke-keys"
Jul 17 11:37:05.099280 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:05.099179    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a92901a5-77cd-4d34-b6f8-85b43bcb99c1" containerName="revoke-keys"
Jul 17 11:37:05.102010 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:05.101976    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-r7fk7"
Jul 17 11:37:05.125518 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:05.125491    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-r7fk7"]
Jul 17 11:37:05.213974 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:05.213862    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a8c0669-7cc5-4b8e-8b97-a584b2339ffd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-57741a73-r7fk7\" (UID: \"7a8c0669-7cc5-4b8e-8b97-a584b2339ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-r7fk7"
Jul 17 11:37:05.314657 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:05.314607    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a8c0669-7cc5-4b8e-8b97-a584b2339ffd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-57741a73-r7fk7\" (UID: \"7a8c0669-7cc5-4b8e-8b97-a584b2339ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-r7fk7"
Jul 17 11:37:05.315242 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:05.315226    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a8c0669-7cc5-4b8e-8b97-a584b2339ffd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-57741a73-r7fk7\" (UID: \"7a8c0669-7cc5-4b8e-8b97-a584b2339ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-r7fk7"
Jul 17 11:37:05.410969 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:05.410938    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-r7fk7"
Jul 17 11:37:05.545151 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:05.545120    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-6t2pz" event={"ID":"a92901a5-77cd-4d34-b6f8-85b43bcb99c1","Type":"ContainerDied","Data":"e52f1418b9a33bbf9c08fc625a2ff6834a1c6f8765845da06dd7aae70e09af35"}
Jul 17 11:37:05.545310 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:05.545181    2563 scope.go:117] "RemoveContainer" containerID="f6d1deeb332e7b5c508f047ce52b834a14e07b24f0a2aa336423a0346a1fd65e"
Jul 17 11:37:05.545310 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:05.545136    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-6t2pz"
Jul 17 11:37:05.553020 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:05.552999    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-r7fk7"]
Jul 17 11:37:05.556005 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:37:05.555983    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a8c0669_7cc5_4b8e_8b97_a584b2339ffd.slice/crio-62f20de600123e7392d0f05e1383264fb678590ac8a1a4416c9d3a101cbc7ee0 WatchSource:0}: Error finding container 62f20de600123e7392d0f05e1383264fb678590ac8a1a4416c9d3a101cbc7ee0: Status 404 returned error can't find the container with id 62f20de600123e7392d0f05e1383264fb678590ac8a1a4416c9d3a101cbc7ee0
Jul 17 11:37:05.584468 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:05.584446    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-6t2pz"]
Jul 17 11:37:05.593641 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:05.593621    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-6t2pz"]
Jul 17 11:37:05.666395 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:05.666304    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a92901a5-77cd-4d34-b6f8-85b43bcb99c1" path="/var/lib/kubelet/pods/a92901a5-77cd-4d34-b6f8-85b43bcb99c1/volumes"
Jul 17 11:37:06.358299 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:06.358257    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:37:06.522503 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:06.522420    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:37:06.551944 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:06.551915    2563 generic.go:356] "Generic (PLEG): container finished" podID="7a8c0669-7cc5-4b8e-8b97-a584b2339ffd" containerID="720c899c90b542610f699bf6e678ea70f244baa181eef542b8a2ef517f3c1b01" exitCode=7
Jul 17 11:37:06.552102 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:06.551974    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-r7fk7" event={"ID":"7a8c0669-7cc5-4b8e-8b97-a584b2339ffd","Type":"ContainerDied","Data":"720c899c90b542610f699bf6e678ea70f244baa181eef542b8a2ef517f3c1b01"}
Jul 17 11:37:06.552102 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:06.552025    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-r7fk7" event={"ID":"7a8c0669-7cc5-4b8e-8b97-a584b2339ffd","Type":"ContainerStarted","Data":"62f20de600123e7392d0f05e1383264fb678590ac8a1a4416c9d3a101cbc7ee0"}
Jul 17 11:37:06.552251 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:06.552237    2563 scope.go:117] "RemoveContainer" containerID="720c899c90b542610f699bf6e678ea70f244baa181eef542b8a2ef517f3c1b01"
Jul 17 11:37:07.558723 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:07.558642    2563 generic.go:356] "Generic (PLEG): container finished" podID="7a8c0669-7cc5-4b8e-8b97-a584b2339ffd" containerID="bf3f365b343e947d9d8268c5be0af7d358ba7d9cc59fde855ca54221814fafd5" exitCode=7
Jul 17 11:37:07.558723 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:07.558719    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-r7fk7" event={"ID":"7a8c0669-7cc5-4b8e-8b97-a584b2339ffd","Type":"ContainerDied","Data":"bf3f365b343e947d9d8268c5be0af7d358ba7d9cc59fde855ca54221814fafd5"}
Jul 17 11:37:07.559148 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:07.558764    2563 scope.go:117] "RemoveContainer" containerID="720c899c90b542610f699bf6e678ea70f244baa181eef542b8a2ef517f3c1b01"
Jul 17 11:37:07.559148 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:07.558939    2563 scope.go:117] "RemoveContainer" containerID="bf3f365b343e947d9d8268c5be0af7d358ba7d9cc59fde855ca54221814fafd5"
Jul 17 11:37:07.559248 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:37:07.559182    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-57741a73-r7fk7_odh-ai-gateway-infra(7a8c0669-7cc5-4b8e-8b97-a584b2339ffd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-r7fk7" podUID="7a8c0669-7cc5-4b8e-8b97-a584b2339ffd"
Jul 17 11:37:08.565684 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:08.565657    2563 scope.go:117] "RemoveContainer" containerID="bf3f365b343e947d9d8268c5be0af7d358ba7d9cc59fde855ca54221814fafd5"
Jul 17 11:37:08.566094 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:37:08.565830    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-57741a73-r7fk7_odh-ai-gateway-infra(7a8c0669-7cc5-4b8e-8b97-a584b2339ffd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-r7fk7" podUID="7a8c0669-7cc5-4b8e-8b97-a584b2339ffd"
Jul 17 11:37:21.660312 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:21.660279    2563 scope.go:117] "RemoveContainer" containerID="bf3f365b343e947d9d8268c5be0af7d358ba7d9cc59fde855ca54221814fafd5"
Jul 17 11:37:22.633694 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:22.633659    2563 generic.go:356] "Generic (PLEG): container finished" podID="7a8c0669-7cc5-4b8e-8b97-a584b2339ffd" containerID="c4a392536f489dd20d120eb4b34b927bdeba75c5c093eb5396aeb0e8bcebe616" exitCode=7
Jul 17 11:37:22.633694 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:22.633697    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-r7fk7" event={"ID":"7a8c0669-7cc5-4b8e-8b97-a584b2339ffd","Type":"ContainerDied","Data":"c4a392536f489dd20d120eb4b34b927bdeba75c5c093eb5396aeb0e8bcebe616"}
Jul 17 11:37:22.633955 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:22.633745    2563 scope.go:117] "RemoveContainer" containerID="bf3f365b343e947d9d8268c5be0af7d358ba7d9cc59fde855ca54221814fafd5"
Jul 17 11:37:22.634127 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:22.634105    2563 scope.go:117] "RemoveContainer" containerID="c4a392536f489dd20d120eb4b34b927bdeba75c5c093eb5396aeb0e8bcebe616"
Jul 17 11:37:22.634370 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:37:22.634347    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-57741a73-r7fk7_odh-ai-gateway-infra(7a8c0669-7cc5-4b8e-8b97-a584b2339ffd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-r7fk7" podUID="7a8c0669-7cc5-4b8e-8b97-a584b2339ffd"
Jul 17 11:37:23.675327 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:23.675294    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-r7fk7"]
Jul 17 11:37:23.807312 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:23.807287    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-r7fk7"
Jul 17 11:37:23.867390 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:23.867354    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a8c0669-7cc5-4b8e-8b97-a584b2339ffd-maas-api-service-ca\") pod \"7a8c0669-7cc5-4b8e-8b97-a584b2339ffd\" (UID: \"7a8c0669-7cc5-4b8e-8b97-a584b2339ffd\") "
Jul 17 11:37:23.867708 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:23.867683    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a8c0669-7cc5-4b8e-8b97-a584b2339ffd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7a8c0669-7cc5-4b8e-8b97-a584b2339ffd" (UID: "7a8c0669-7cc5-4b8e-8b97-a584b2339ffd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:37:23.968734 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:23.968637    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a8c0669-7cc5-4b8e-8b97-a584b2339ffd-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:37:24.648045 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:24.648009    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-r7fk7"
Jul 17 11:37:24.648209 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:24.648050    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-r7fk7" event={"ID":"7a8c0669-7cc5-4b8e-8b97-a584b2339ffd","Type":"ContainerDied","Data":"62f20de600123e7392d0f05e1383264fb678590ac8a1a4416c9d3a101cbc7ee0"}
Jul 17 11:37:24.648209 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:24.648121    2563 scope.go:117] "RemoveContainer" containerID="c4a392536f489dd20d120eb4b34b927bdeba75c5c093eb5396aeb0e8bcebe616"
Jul 17 11:37:24.680437 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:24.680407    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-r7fk7"]
Jul 17 11:37:24.684431 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:24.684411    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-r7fk7"]
Jul 17 11:37:25.286639 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:25.286609    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:37:25.424493 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:25.424432    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:37:25.668845 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:25.668809    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a8c0669-7cc5-4b8e-8b97-a584b2339ffd" path="/var/lib/kubelet/pods/7a8c0669-7cc5-4b8e-8b97-a584b2339ffd/volumes"
Jul 17 11:37:25.732016 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:25.731989    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nj5mj_55d1de37-8635-4de8-ab59-4ebfd0346a40/ovn-acl-logging/0.log"
Jul 17 11:37:25.737596 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:25.737576    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nj5mj_55d1de37-8635-4de8-ab59-4ebfd0346a40/ovn-acl-logging/0.log"
Jul 17 11:37:32.522613 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:32.522574    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:37:35.513336 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:35.513301    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:37:45.339585 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:45.339551    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:37:45.577327 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:45.577295    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:37:45.935800 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:45.935757    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:37:46.190200 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:46.190106    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:37:46.988878 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:46.988840    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:37:48.098835 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:48.098803    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:37:58.356153 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:58.356122    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:37:58.671552 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:58.671521    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:37:58.871580 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:58.871528    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:37:59.356238 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:37:59.356199    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:38:25.321582 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:38:25.321546    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:38:32.116359 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:38:32.116316    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:38:32.221045 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:38:32.220986    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:38:32.315897 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:38:32.315865    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:38:48.316297 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:38:48.316262    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:38:54.955235 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:38:54.955183    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:38:55.035820 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:38:55.035789    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:38:55.143724 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:38:55.143684    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:05.752658 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:05.752607    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:06.592168 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:06.592131    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:13.117304 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:13.117267    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:13.228008 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:13.227971    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:13.316803 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:13.316758    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:13.448141 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:13.448107    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:14.125012 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:14.124974    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:14.356817 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:14.356781    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:14.486673 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:14.486623    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:14.592739 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:14.592710    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:16.278866 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:16.278837    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:22.725953 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:22.725847    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:22.797763 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:22.797712    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:23.436998 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:23.436959    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:29.926784 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:29.926747    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:30.002691 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:30.002658    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:30.179734 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:30.179652    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:30.539090 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:30.538992    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:31.314666 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:31.314626    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:34.054949 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:34.054909    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:34.160879 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:34.160844    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:34.303708 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:34.303679    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:34.458672 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:34.458637    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:34.869273 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:34.869165    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:34.969660 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:34.969614    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:38.491371 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:38.491337    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:39.210018 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:39.209985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:39.307663 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:39.307631    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:39.659871 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:39.659841    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:45.313671 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:45.313639    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-tcd28"]
Jul 17 11:39:45.314064 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:45.313976    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a8c0669-7cc5-4b8e-8b97-a584b2339ffd" containerName="revoke-keys"
Jul 17 11:39:45.314064 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:45.313988    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a8c0669-7cc5-4b8e-8b97-a584b2339ffd" containerName="revoke-keys"
Jul 17 11:39:45.314064 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:45.314006    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a8c0669-7cc5-4b8e-8b97-a584b2339ffd" containerName="revoke-keys"
Jul 17 11:39:45.314064 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:45.314012    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a8c0669-7cc5-4b8e-8b97-a584b2339ffd" containerName="revoke-keys"
Jul 17 11:39:45.314064 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:45.314078    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a92901a5-77cd-4d34-b6f8-85b43bcb99c1" containerName="revoke-keys"
Jul 17 11:39:45.314258 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:45.314087    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a8c0669-7cc5-4b8e-8b97-a584b2339ffd" containerName="revoke-keys"
Jul 17 11:39:45.314258 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:45.314095    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a8c0669-7cc5-4b8e-8b97-a584b2339ffd" containerName="revoke-keys"
Jul 17 11:39:45.314258 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:45.314103    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a8c0669-7cc5-4b8e-8b97-a584b2339ffd" containerName="revoke-keys"
Jul 17 11:39:45.317176 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:45.317157    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-tcd28"
Jul 17 11:39:45.319757 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:45.319737    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 11:39:45.319875 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:45.319797    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwv2h\""
Jul 17 11:39:45.326056 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:45.326036    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-tcd28"]
Jul 17 11:39:45.423226 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:45.423198    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/195d3924-d376-4cba-bf20-be55a78a0234-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-57741a73-tcd28\" (UID: \"195d3924-d376-4cba-bf20-be55a78a0234\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-tcd28"
Jul 17 11:39:45.523989 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:45.523953    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/195d3924-d376-4cba-bf20-be55a78a0234-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-57741a73-tcd28\" (UID: \"195d3924-d376-4cba-bf20-be55a78a0234\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-tcd28"
Jul 17 11:39:45.524575 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:45.524556    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/195d3924-d376-4cba-bf20-be55a78a0234-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-57741a73-tcd28\" (UID: \"195d3924-d376-4cba-bf20-be55a78a0234\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-tcd28"
Jul 17 11:39:45.628190 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:45.628113    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-tcd28"
Jul 17 11:39:45.758937 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:45.758908    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-tcd28"]
Jul 17 11:39:45.760669 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:39:45.760644    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195d3924_d376_4cba_bf20_be55a78a0234.slice/crio-8334ae494a9070b1477978ed90649dcb9f44f1820bdf31ab4e6f8aa1e2a87093 WatchSource:0}: Error finding container 8334ae494a9070b1477978ed90649dcb9f44f1820bdf31ab4e6f8aa1e2a87093: Status 404 returned error can't find the container with id 8334ae494a9070b1477978ed90649dcb9f44f1820bdf31ab4e6f8aa1e2a87093
Jul 17 11:39:46.027473 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:46.027436    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:46.126868 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:46.126837    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:39:46.353669 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:46.353584    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-tcd28" event={"ID":"195d3924-d376-4cba-bf20-be55a78a0234","Type":"ContainerStarted","Data":"7aae1fc9bf6648559c0248ec5842398a955c3a87d4e36274cf556168356fc885"}
Jul 17 11:39:46.353669 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:46.353631    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-tcd28" event={"ID":"195d3924-d376-4cba-bf20-be55a78a0234","Type":"ContainerStarted","Data":"8334ae494a9070b1477978ed90649dcb9f44f1820bdf31ab4e6f8aa1e2a87093"}
Jul 17 11:39:46.375490 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:46.375439    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-tcd28" podStartSLOduration=1.375424142 podStartE2EDuration="1.375424142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:39:46.373732509 +0000 UTC m=+2541.303272153" watchObservedRunningTime="2026-07-17 11:39:46.375424142 +0000 UTC m=+2541.304963783"
Jul 17 11:39:47.360263 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:47.360231    2563 generic.go:356] "Generic (PLEG): container finished" podID="195d3924-d376-4cba-bf20-be55a78a0234" containerID="7aae1fc9bf6648559c0248ec5842398a955c3a87d4e36274cf556168356fc885" exitCode=6
Jul 17 11:39:47.360660 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:47.360318    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-tcd28" event={"ID":"195d3924-d376-4cba-bf20-be55a78a0234","Type":"ContainerDied","Data":"7aae1fc9bf6648559c0248ec5842398a955c3a87d4e36274cf556168356fc885"}
Jul 17 11:39:47.360660 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:47.360472    2563 scope.go:117] "RemoveContainer" containerID="7aae1fc9bf6648559c0248ec5842398a955c3a87d4e36274cf556168356fc885"
Jul 17 11:39:48.367192 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:48.367100    2563 generic.go:356] "Generic (PLEG): container finished" podID="195d3924-d376-4cba-bf20-be55a78a0234" containerID="103bfa5067fb4498528768f7c172d6a2e9f4d6f43e0c2faf5fe8f66dbe4d9bab" exitCode=6
Jul 17 11:39:48.367599 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:48.367178    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-tcd28" event={"ID":"195d3924-d376-4cba-bf20-be55a78a0234","Type":"ContainerDied","Data":"103bfa5067fb4498528768f7c172d6a2e9f4d6f43e0c2faf5fe8f66dbe4d9bab"}
Jul 17 11:39:48.367599 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:48.367245    2563 scope.go:117] "RemoveContainer" containerID="7aae1fc9bf6648559c0248ec5842398a955c3a87d4e36274cf556168356fc885"
Jul 17 11:39:48.367599 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:48.367433    2563 scope.go:117] "RemoveContainer" containerID="103bfa5067fb4498528768f7c172d6a2e9f4d6f43e0c2faf5fe8f66dbe4d9bab"
Jul 17 11:39:48.367709 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:39:48.367678    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-57741a73-tcd28_odh-ai-gateway-infra(195d3924-d376-4cba-bf20-be55a78a0234)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-tcd28" podUID="195d3924-d376-4cba-bf20-be55a78a0234"
Jul 17 11:39:49.373911 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:49.373879    2563 scope.go:117] "RemoveContainer" containerID="103bfa5067fb4498528768f7c172d6a2e9f4d6f43e0c2faf5fe8f66dbe4d9bab"
Jul 17 11:39:49.374341 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:39:49.374057    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-57741a73-tcd28_odh-ai-gateway-infra(195d3924-d376-4cba-bf20-be55a78a0234)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-tcd28" podUID="195d3924-d376-4cba-bf20-be55a78a0234"
Jul 17 11:39:56.955530 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:39:56.955492    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:40:01.571116 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:01.571057    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:40:03.659713 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:03.659677    2563 scope.go:117] "RemoveContainer" containerID="103bfa5067fb4498528768f7c172d6a2e9f4d6f43e0c2faf5fe8f66dbe4d9bab"
Jul 17 11:40:04.447378 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:04.447347    2563 generic.go:356] "Generic (PLEG): container finished" podID="195d3924-d376-4cba-bf20-be55a78a0234" containerID="569310acb4084337db77c43da6c55e1b4b9adf1289fb88533746266ceb24a7c3" exitCode=6
Jul 17 11:40:04.447551 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:04.447415    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-tcd28" event={"ID":"195d3924-d376-4cba-bf20-be55a78a0234","Type":"ContainerDied","Data":"569310acb4084337db77c43da6c55e1b4b9adf1289fb88533746266ceb24a7c3"}
Jul 17 11:40:04.447551 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:04.447475    2563 scope.go:117] "RemoveContainer" containerID="103bfa5067fb4498528768f7c172d6a2e9f4d6f43e0c2faf5fe8f66dbe4d9bab"
Jul 17 11:40:04.447764 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:04.447750    2563 scope.go:117] "RemoveContainer" containerID="569310acb4084337db77c43da6c55e1b4b9adf1289fb88533746266ceb24a7c3"
Jul 17 11:40:04.448021 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:40:04.448000    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-57741a73-tcd28_odh-ai-gateway-infra(195d3924-d376-4cba-bf20-be55a78a0234)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-tcd28" podUID="195d3924-d376-4cba-bf20-be55a78a0234"
Jul 17 11:40:05.479258 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:05.479222    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-tcd28"]
Jul 17 11:40:05.614715 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:05.614689    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-tcd28"
Jul 17 11:40:05.699542 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:05.699511    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/195d3924-d376-4cba-bf20-be55a78a0234-maas-api-service-ca\") pod \"195d3924-d376-4cba-bf20-be55a78a0234\" (UID: \"195d3924-d376-4cba-bf20-be55a78a0234\") "
Jul 17 11:40:05.699876 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:05.699851    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/195d3924-d376-4cba-bf20-be55a78a0234-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "195d3924-d376-4cba-bf20-be55a78a0234" (UID: "195d3924-d376-4cba-bf20-be55a78a0234"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:40:05.800698 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:05.800627    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/195d3924-d376-4cba-bf20-be55a78a0234-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:40:06.465288 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:06.465260    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-tcd28"
Jul 17 11:40:06.465467 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:06.465298    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-tcd28" event={"ID":"195d3924-d376-4cba-bf20-be55a78a0234","Type":"ContainerDied","Data":"8334ae494a9070b1477978ed90649dcb9f44f1820bdf31ab4e6f8aa1e2a87093"}
Jul 17 11:40:06.465467 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:06.465355    2563 scope.go:117] "RemoveContainer" containerID="569310acb4084337db77c43da6c55e1b4b9adf1289fb88533746266ceb24a7c3"
Jul 17 11:40:06.491751 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:06.491711    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-tcd28"]
Jul 17 11:40:06.493856 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:06.493836    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-tcd28"]
Jul 17 11:40:07.665044 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:07.665013    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="195d3924-d376-4cba-bf20-be55a78a0234" path="/var/lib/kubelet/pods/195d3924-d376-4cba-bf20-be55a78a0234/volumes"
Jul 17 11:40:08.219751 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:08.219718    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:40:08.323143 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:08.323108    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:40:12.285114 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:12.285064    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:40:12.527461 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:12.527432    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:40:12.661904 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:12.661862    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:40:12.760732 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:12.760689    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:40:12.914648 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:12.914542    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:40:19.373286 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:19.373222    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:40:19.474745 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:19.474715    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:40:19.579216 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:19.579187    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:40:19.762730 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:19.762696    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:40:26.135301 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:26.135262    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:40:26.253190 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:26.253155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:40:33.084485 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:33.084449    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:40:33.429125 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:33.429088    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:40:33.529569 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:33.529519    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:40:36.234524 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:36.234486    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:40:36.337321 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:36.337287    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:40:36.432130 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:36.432091    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:40:36.533386 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:36.533295    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:40:37.013979 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:37.013936    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:40:37.113520 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:37.113482    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:40:40.783881 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:40.783822    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:40:54.332918 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:54.332830    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:40:57.481975 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:57.481934    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-7bdg6"]
Jul 17 11:40:57.482400 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:57.482311    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="195d3924-d376-4cba-bf20-be55a78a0234" containerName="revoke-keys"
Jul 17 11:40:57.482400 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:57.482322    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="195d3924-d376-4cba-bf20-be55a78a0234" containerName="revoke-keys"
Jul 17 11:40:57.482400 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:57.482333    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a8c0669-7cc5-4b8e-8b97-a584b2339ffd" containerName="revoke-keys"
Jul 17 11:40:57.482400 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:57.482338    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a8c0669-7cc5-4b8e-8b97-a584b2339ffd" containerName="revoke-keys"
Jul 17 11:40:57.482400 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:57.482353    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="195d3924-d376-4cba-bf20-be55a78a0234" containerName="revoke-keys"
Jul 17 11:40:57.482400 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:57.482359    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="195d3924-d376-4cba-bf20-be55a78a0234" containerName="revoke-keys"
Jul 17 11:40:57.482628 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:57.482406    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="195d3924-d376-4cba-bf20-be55a78a0234" containerName="revoke-keys"
Jul 17 11:40:57.482628 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:57.482417    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="195d3924-d376-4cba-bf20-be55a78a0234" containerName="revoke-keys"
Jul 17 11:40:57.485347 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:57.485332    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-7bdg6"
Jul 17 11:40:57.488347 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:57.488325    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwv2h\""
Jul 17 11:40:57.488485 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:57.488362    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 11:40:57.497119 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:57.497096    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-7bdg6"]
Jul 17 11:40:57.556661 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:57.556629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/442b5e06-c026-4663-8bfc-cbb1555c05ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45989be5-7bdg6\" (UID: \"442b5e06-c026-4663-8bfc-cbb1555c05ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-7bdg6"
Jul 17 11:40:57.657972 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:57.657917    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/442b5e06-c026-4663-8bfc-cbb1555c05ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45989be5-7bdg6\" (UID: \"442b5e06-c026-4663-8bfc-cbb1555c05ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-7bdg6"
Jul 17 11:40:57.658565 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:57.658546    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/442b5e06-c026-4663-8bfc-cbb1555c05ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45989be5-7bdg6\" (UID: \"442b5e06-c026-4663-8bfc-cbb1555c05ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-7bdg6"
Jul 17 11:40:57.796162 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:57.796058    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-7bdg6"
Jul 17 11:40:57.927122 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:57.927093    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-7bdg6"]
Jul 17 11:40:57.929119 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:40:57.929092    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442b5e06_c026_4663_8bfc_cbb1555c05ca.slice/crio-bace019dd7d6af275e45d2fd67bae8e635f526b97c377046f1d727d46fd241c1 WatchSource:0}: Error finding container bace019dd7d6af275e45d2fd67bae8e635f526b97c377046f1d727d46fd241c1: Status 404 returned error can't find the container with id bace019dd7d6af275e45d2fd67bae8e635f526b97c377046f1d727d46fd241c1
Jul 17 11:40:58.723327 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:58.723293    2563 generic.go:356] "Generic (PLEG): container finished" podID="442b5e06-c026-4663-8bfc-cbb1555c05ca" containerID="477977dff526ce708e3c80f50cf009032867db72d1da661f6caa398f785dc51d" exitCode=6
Jul 17 11:40:58.723695 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:58.723375    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-7bdg6" event={"ID":"442b5e06-c026-4663-8bfc-cbb1555c05ca","Type":"ContainerDied","Data":"477977dff526ce708e3c80f50cf009032867db72d1da661f6caa398f785dc51d"}
Jul 17 11:40:58.723695 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:58.723425    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-7bdg6" event={"ID":"442b5e06-c026-4663-8bfc-cbb1555c05ca","Type":"ContainerStarted","Data":"bace019dd7d6af275e45d2fd67bae8e635f526b97c377046f1d727d46fd241c1"}
Jul 17 11:40:58.723695 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:58.723512    2563 scope.go:117] "RemoveContainer" containerID="477977dff526ce708e3c80f50cf009032867db72d1da661f6caa398f785dc51d"
Jul 17 11:40:59.731664 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:59.731572    2563 generic.go:356] "Generic (PLEG): container finished" podID="442b5e06-c026-4663-8bfc-cbb1555c05ca" containerID="707601ad92e2a58fe8651e42228b46ed43f410cbed46d653d7601433e89f216a" exitCode=6
Jul 17 11:40:59.731664 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:59.731611    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-7bdg6" event={"ID":"442b5e06-c026-4663-8bfc-cbb1555c05ca","Type":"ContainerDied","Data":"707601ad92e2a58fe8651e42228b46ed43f410cbed46d653d7601433e89f216a"}
Jul 17 11:40:59.731664 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:59.731655    2563 scope.go:117] "RemoveContainer" containerID="477977dff526ce708e3c80f50cf009032867db72d1da661f6caa398f785dc51d"
Jul 17 11:40:59.732248 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:40:59.731891    2563 scope.go:117] "RemoveContainer" containerID="707601ad92e2a58fe8651e42228b46ed43f410cbed46d653d7601433e89f216a"
Jul 17 11:40:59.732248 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:40:59.732162    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45989be5-7bdg6_odh-ai-gateway-infra(442b5e06-c026-4663-8bfc-cbb1555c05ca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-7bdg6" podUID="442b5e06-c026-4663-8bfc-cbb1555c05ca"
Jul 17 11:41:00.738368 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:00.738337    2563 scope.go:117] "RemoveContainer" containerID="707601ad92e2a58fe8651e42228b46ed43f410cbed46d653d7601433e89f216a"
Jul 17 11:41:00.738750 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:41:00.738515    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45989be5-7bdg6_odh-ai-gateway-infra(442b5e06-c026-4663-8bfc-cbb1555c05ca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-7bdg6" podUID="442b5e06-c026-4663-8bfc-cbb1555c05ca"
Jul 17 11:41:11.659879 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:11.659852    2563 scope.go:117] "RemoveContainer" containerID="707601ad92e2a58fe8651e42228b46ed43f410cbed46d653d7601433e89f216a"
Jul 17 11:41:12.806463 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:12.806425    2563 generic.go:356] "Generic (PLEG): container finished" podID="442b5e06-c026-4663-8bfc-cbb1555c05ca" containerID="8ca98931c74e5e9663ada1b18a799357a2486e3c669a81184aa02b92957d9ec7" exitCode=6
Jul 17 11:41:12.806903 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:12.806491    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-7bdg6" event={"ID":"442b5e06-c026-4663-8bfc-cbb1555c05ca","Type":"ContainerDied","Data":"8ca98931c74e5e9663ada1b18a799357a2486e3c669a81184aa02b92957d9ec7"}
Jul 17 11:41:12.806903 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:12.806537    2563 scope.go:117] "RemoveContainer" containerID="707601ad92e2a58fe8651e42228b46ed43f410cbed46d653d7601433e89f216a"
Jul 17 11:41:12.806903 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:12.806813    2563 scope.go:117] "RemoveContainer" containerID="8ca98931c74e5e9663ada1b18a799357a2486e3c669a81184aa02b92957d9ec7"
Jul 17 11:41:12.807031 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:41:12.807002    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45989be5-7bdg6_odh-ai-gateway-infra(442b5e06-c026-4663-8bfc-cbb1555c05ca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-7bdg6" podUID="442b5e06-c026-4663-8bfc-cbb1555c05ca"
Jul 17 11:41:13.838627 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:13.838593    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-7bdg6"]
Jul 17 11:41:13.970720 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:13.970698    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-7bdg6"
Jul 17 11:41:14.006234 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:14.006195    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/442b5e06-c026-4663-8bfc-cbb1555c05ca-maas-api-service-ca\") pod \"442b5e06-c026-4663-8bfc-cbb1555c05ca\" (UID: \"442b5e06-c026-4663-8bfc-cbb1555c05ca\") "
Jul 17 11:41:14.006500 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:14.006480    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/442b5e06-c026-4663-8bfc-cbb1555c05ca-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "442b5e06-c026-4663-8bfc-cbb1555c05ca" (UID: "442b5e06-c026-4663-8bfc-cbb1555c05ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:41:14.106999 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:14.106907    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/442b5e06-c026-4663-8bfc-cbb1555c05ca-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:41:14.819063 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:14.819035    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-7bdg6"
Jul 17 11:41:14.819251 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:14.819093    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-7bdg6" event={"ID":"442b5e06-c026-4663-8bfc-cbb1555c05ca","Type":"ContainerDied","Data":"bace019dd7d6af275e45d2fd67bae8e635f526b97c377046f1d727d46fd241c1"}
Jul 17 11:41:14.819251 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:14.819156    2563 scope.go:117] "RemoveContainer" containerID="8ca98931c74e5e9663ada1b18a799357a2486e3c669a81184aa02b92957d9ec7"
Jul 17 11:41:14.844418 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:14.844394    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-7bdg6"]
Jul 17 11:41:14.851700 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:14.851676    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-7bdg6"]
Jul 17 11:41:15.665797 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:15.665765    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="442b5e06-c026-4663-8bfc-cbb1555c05ca" path="/var/lib/kubelet/pods/442b5e06-c026-4663-8bfc-cbb1555c05ca/volumes"
Jul 17 11:41:19.239116 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:19.239082    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63359af2-7gpnl"]
Jul 17 11:41:19.239515 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:19.239393    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="442b5e06-c026-4663-8bfc-cbb1555c05ca" containerName="revoke-keys"
Jul 17 11:41:19.239515 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:19.239416    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="442b5e06-c026-4663-8bfc-cbb1555c05ca" containerName="revoke-keys"
Jul 17 11:41:19.239515 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:19.239434    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="195d3924-d376-4cba-bf20-be55a78a0234" containerName="revoke-keys"
Jul 17 11:41:19.239515 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:19.239439    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="195d3924-d376-4cba-bf20-be55a78a0234" containerName="revoke-keys"
Jul 17 11:41:19.239515 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:19.239450    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="442b5e06-c026-4663-8bfc-cbb1555c05ca" containerName="revoke-keys"
Jul 17 11:41:19.239515 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:19.239455    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="442b5e06-c026-4663-8bfc-cbb1555c05ca" containerName="revoke-keys"
Jul 17 11:41:19.239515 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:19.239504    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="442b5e06-c026-4663-8bfc-cbb1555c05ca" containerName="revoke-keys"
Jul 17 11:41:19.239515 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:19.239512    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="442b5e06-c026-4663-8bfc-cbb1555c05ca" containerName="revoke-keys"
Jul 17 11:41:19.239515 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:19.239520    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="195d3924-d376-4cba-bf20-be55a78a0234" containerName="revoke-keys"
Jul 17 11:41:19.243723 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:19.243706    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63359af2-7gpnl"
Jul 17 11:41:19.246511 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:19.246490    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwv2h\""
Jul 17 11:41:19.246617 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:19.246532    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 11:41:19.251052 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:19.250766    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63359af2-7gpnl"]
Jul 17 11:41:19.345696 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:19.345666    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51fb0c2f-1075-47ee-93ce-5dc80356aab5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-63359af2-7gpnl\" (UID: \"51fb0c2f-1075-47ee-93ce-5dc80356aab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63359af2-7gpnl"
Jul 17 11:41:19.446426 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:19.446395    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51fb0c2f-1075-47ee-93ce-5dc80356aab5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-63359af2-7gpnl\" (UID: \"51fb0c2f-1075-47ee-93ce-5dc80356aab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63359af2-7gpnl"
Jul 17 11:41:19.447022 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:19.446999    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51fb0c2f-1075-47ee-93ce-5dc80356aab5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-63359af2-7gpnl\" (UID: \"51fb0c2f-1075-47ee-93ce-5dc80356aab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63359af2-7gpnl"
Jul 17 11:41:19.555573 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:19.555504    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63359af2-7gpnl"
Jul 17 11:41:19.692693 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:19.692666    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63359af2-7gpnl"]
Jul 17 11:41:19.694700 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:41:19.694672    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fb0c2f_1075_47ee_93ce_5dc80356aab5.slice/crio-08ad9420221b13adbf34760c03dd8c5f911c10c102a2e47233734443a48cf8a9 WatchSource:0}: Error finding container 08ad9420221b13adbf34760c03dd8c5f911c10c102a2e47233734443a48cf8a9: Status 404 returned error can't find the container with id 08ad9420221b13adbf34760c03dd8c5f911c10c102a2e47233734443a48cf8a9
Jul 17 11:41:19.844495 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:19.844408    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63359af2-7gpnl" event={"ID":"51fb0c2f-1075-47ee-93ce-5dc80356aab5","Type":"ContainerStarted","Data":"08ad9420221b13adbf34760c03dd8c5f911c10c102a2e47233734443a48cf8a9"}
Jul 17 11:41:20.850417 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:20.850378    2563 generic.go:356] "Generic (PLEG): container finished" podID="51fb0c2f-1075-47ee-93ce-5dc80356aab5" containerID="9a93cc0031b002e06cb73b489a68ce4a4634bd1e976087fe88dac10c24b75fda" exitCode=0
Jul 17 11:41:20.850814 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:20.850465    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63359af2-7gpnl" event={"ID":"51fb0c2f-1075-47ee-93ce-5dc80356aab5","Type":"ContainerDied","Data":"9a93cc0031b002e06cb73b489a68ce4a4634bd1e976087fe88dac10c24b75fda"}
Jul 17 11:41:21.993227 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:21.993206    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63359af2-7gpnl"
Jul 17 11:41:22.070446 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:22.070412    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51fb0c2f-1075-47ee-93ce-5dc80356aab5-maas-api-service-ca\") pod \"51fb0c2f-1075-47ee-93ce-5dc80356aab5\" (UID: \"51fb0c2f-1075-47ee-93ce-5dc80356aab5\") "
Jul 17 11:41:22.070806 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:22.070778    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51fb0c2f-1075-47ee-93ce-5dc80356aab5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "51fb0c2f-1075-47ee-93ce-5dc80356aab5" (UID: "51fb0c2f-1075-47ee-93ce-5dc80356aab5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:41:22.171919 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:22.171886    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51fb0c2f-1075-47ee-93ce-5dc80356aab5-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:41:22.861859 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:22.861830    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63359af2-7gpnl"
Jul 17 11:41:22.862086 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:22.861830    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63359af2-7gpnl" event={"ID":"51fb0c2f-1075-47ee-93ce-5dc80356aab5","Type":"ContainerDied","Data":"08ad9420221b13adbf34760c03dd8c5f911c10c102a2e47233734443a48cf8a9"}
Jul 17 11:41:22.862086 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:22.861962    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08ad9420221b13adbf34760c03dd8c5f911c10c102a2e47233734443a48cf8a9"
Jul 17 11:41:29.892545 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:29.892507    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:41:34.340708 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:34.340659    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:41:39.417850 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:39.417811    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:41:45.419608 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:45.419566    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-8sn9v"]
Jul 17 11:41:45.420099 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:45.420057    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="442b5e06-c026-4663-8bfc-cbb1555c05ca" containerName="revoke-keys"
Jul 17 11:41:45.420099 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:45.420092    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="442b5e06-c026-4663-8bfc-cbb1555c05ca" containerName="revoke-keys"
Jul 17 11:41:45.420228 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:45.420106    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51fb0c2f-1075-47ee-93ce-5dc80356aab5" containerName="revoke-keys"
Jul 17 11:41:45.420228 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:45.420115    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="51fb0c2f-1075-47ee-93ce-5dc80356aab5" containerName="revoke-keys"
Jul 17 11:41:45.420228 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:45.420221    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="442b5e06-c026-4663-8bfc-cbb1555c05ca" containerName="revoke-keys"
Jul 17 11:41:45.420380 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:45.420235    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="51fb0c2f-1075-47ee-93ce-5dc80356aab5" containerName="revoke-keys"
Jul 17 11:41:45.423113 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:45.423094    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-8sn9v"
Jul 17 11:41:45.426213 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:45.426187    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwv2h\""
Jul 17 11:41:45.426320 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:45.426220    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 11:41:45.433394 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:45.433373    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-8sn9v"]
Jul 17 11:41:45.581045 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:45.581002    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85c3cc4d-ee09-424b-9e1d-306b8bce10a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-57741a73-8sn9v\" (UID: \"85c3cc4d-ee09-424b-9e1d-306b8bce10a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-8sn9v"
Jul 17 11:41:45.682158 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:45.682060    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85c3cc4d-ee09-424b-9e1d-306b8bce10a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-57741a73-8sn9v\" (UID: \"85c3cc4d-ee09-424b-9e1d-306b8bce10a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-8sn9v"
Jul 17 11:41:45.682701 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:45.682679    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85c3cc4d-ee09-424b-9e1d-306b8bce10a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-57741a73-8sn9v\" (UID: \"85c3cc4d-ee09-424b-9e1d-306b8bce10a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-8sn9v"
Jul 17 11:41:45.733126 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:45.733089    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-8sn9v"
Jul 17 11:41:45.863893 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:45.863868    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-8sn9v"]
Jul 17 11:41:45.865755 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:41:45.865732    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c3cc4d_ee09_424b_9e1d_306b8bce10a2.slice/crio-c702725addd3c06b7a98647cc6178c00c8242abc1b523af89e7f0a186b1059cb WatchSource:0}: Error finding container c702725addd3c06b7a98647cc6178c00c8242abc1b523af89e7f0a186b1059cb: Status 404 returned error can't find the container with id c702725addd3c06b7a98647cc6178c00c8242abc1b523af89e7f0a186b1059cb
Jul 17 11:41:45.980657 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:45.980565    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-8sn9v" event={"ID":"85c3cc4d-ee09-424b-9e1d-306b8bce10a2","Type":"ContainerStarted","Data":"c702725addd3c06b7a98647cc6178c00c8242abc1b523af89e7f0a186b1059cb"}
Jul 17 11:41:46.987666 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:46.987631    2563 generic.go:356] "Generic (PLEG): container finished" podID="85c3cc4d-ee09-424b-9e1d-306b8bce10a2" containerID="4b84330241d985c3bff55c45bb98f3ab3a4bbd2c60b77887cb23ab4f15526350" exitCode=6
Jul 17 11:41:46.988058 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:46.987700    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-8sn9v" event={"ID":"85c3cc4d-ee09-424b-9e1d-306b8bce10a2","Type":"ContainerDied","Data":"4b84330241d985c3bff55c45bb98f3ab3a4bbd2c60b77887cb23ab4f15526350"}
Jul 17 11:41:46.988058 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:46.987928    2563 scope.go:117] "RemoveContainer" containerID="4b84330241d985c3bff55c45bb98f3ab3a4bbd2c60b77887cb23ab4f15526350"
Jul 17 11:41:47.994809 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:47.994776    2563 generic.go:356] "Generic (PLEG): container finished" podID="85c3cc4d-ee09-424b-9e1d-306b8bce10a2" containerID="d10a6edc236ee62a82b28fec4241fdd73fcf94bf623f985904b319e063bb9de9" exitCode=6
Jul 17 11:41:47.995248 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:47.994871    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-8sn9v" event={"ID":"85c3cc4d-ee09-424b-9e1d-306b8bce10a2","Type":"ContainerDied","Data":"d10a6edc236ee62a82b28fec4241fdd73fcf94bf623f985904b319e063bb9de9"}
Jul 17 11:41:47.995248 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:47.994964    2563 scope.go:117] "RemoveContainer" containerID="4b84330241d985c3bff55c45bb98f3ab3a4bbd2c60b77887cb23ab4f15526350"
Jul 17 11:41:47.995248 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:47.995037    2563 scope.go:117] "RemoveContainer" containerID="d10a6edc236ee62a82b28fec4241fdd73fcf94bf623f985904b319e063bb9de9"
Jul 17 11:41:47.995389 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:41:47.995328    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-57741a73-8sn9v_odh-ai-gateway-infra(85c3cc4d-ee09-424b-9e1d-306b8bce10a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-8sn9v" podUID="85c3cc4d-ee09-424b-9e1d-306b8bce10a2"
Jul 17 11:41:49.002777 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:49.002752    2563 scope.go:117] "RemoveContainer" containerID="d10a6edc236ee62a82b28fec4241fdd73fcf94bf623f985904b319e063bb9de9"
Jul 17 11:41:49.003195 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:41:49.002969    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-57741a73-8sn9v_odh-ai-gateway-infra(85c3cc4d-ee09-424b-9e1d-306b8bce10a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-8sn9v" podUID="85c3cc4d-ee09-424b-9e1d-306b8bce10a2"
Jul 17 11:41:52.511420 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:52.511387    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:41:52.608697 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:41:52.608665    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:42:00.659764 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:00.659720    2563 scope.go:117] "RemoveContainer" containerID="d10a6edc236ee62a82b28fec4241fdd73fcf94bf623f985904b319e063bb9de9"
Jul 17 11:42:01.067099 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:01.067004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-8sn9v" event={"ID":"85c3cc4d-ee09-424b-9e1d-306b8bce10a2","Type":"ContainerStarted","Data":"6fc400833cf6a5d4a44f6c049c70775b01d1b9cace960a82a53d81e440e418f5"}
Jul 17 11:42:01.086619 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:01.086565    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-8sn9v" podStartSLOduration=16.086546738 podStartE2EDuration="16.086546738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:42:01.084760845 +0000 UTC m=+2676.014300486" watchObservedRunningTime="2026-07-17 11:42:01.086546738 +0000 UTC m=+2676.016086379"
Jul 17 11:42:02.073788 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:02.073758    2563 generic.go:356] "Generic (PLEG): container finished" podID="85c3cc4d-ee09-424b-9e1d-306b8bce10a2" containerID="6fc400833cf6a5d4a44f6c049c70775b01d1b9cace960a82a53d81e440e418f5" exitCode=6
Jul 17 11:42:02.074241 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:02.073835    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-8sn9v" event={"ID":"85c3cc4d-ee09-424b-9e1d-306b8bce10a2","Type":"ContainerDied","Data":"6fc400833cf6a5d4a44f6c049c70775b01d1b9cace960a82a53d81e440e418f5"}
Jul 17 11:42:02.074241 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:02.073923    2563 scope.go:117] "RemoveContainer" containerID="d10a6edc236ee62a82b28fec4241fdd73fcf94bf623f985904b319e063bb9de9"
Jul 17 11:42:02.074241 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:02.074215    2563 scope.go:117] "RemoveContainer" containerID="6fc400833cf6a5d4a44f6c049c70775b01d1b9cace960a82a53d81e440e418f5"
Jul 17 11:42:02.074456 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:42:02.074437    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-57741a73-8sn9v_odh-ai-gateway-infra(85c3cc4d-ee09-424b-9e1d-306b8bce10a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-8sn9v" podUID="85c3cc4d-ee09-424b-9e1d-306b8bce10a2"
Jul 17 11:42:02.101434 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:02.101403    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-8sn9v"]
Jul 17 11:42:03.209293 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:03.209261    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-8sn9v"
Jul 17 11:42:03.231302 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:03.231270    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85c3cc4d-ee09-424b-9e1d-306b8bce10a2-maas-api-service-ca\") pod \"85c3cc4d-ee09-424b-9e1d-306b8bce10a2\" (UID: \"85c3cc4d-ee09-424b-9e1d-306b8bce10a2\") "
Jul 17 11:42:03.231696 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:03.231670    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85c3cc4d-ee09-424b-9e1d-306b8bce10a2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "85c3cc4d-ee09-424b-9e1d-306b8bce10a2" (UID: "85c3cc4d-ee09-424b-9e1d-306b8bce10a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:42:03.332708 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:03.332623    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85c3cc4d-ee09-424b-9e1d-306b8bce10a2-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:42:04.086382 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:04.086295    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-8sn9v"
Jul 17 11:42:04.086382 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:04.086317    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-8sn9v" event={"ID":"85c3cc4d-ee09-424b-9e1d-306b8bce10a2","Type":"ContainerDied","Data":"c702725addd3c06b7a98647cc6178c00c8242abc1b523af89e7f0a186b1059cb"}
Jul 17 11:42:04.086382 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:04.086373    2563 scope.go:117] "RemoveContainer" containerID="6fc400833cf6a5d4a44f6c049c70775b01d1b9cace960a82a53d81e440e418f5"
Jul 17 11:42:04.112033 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:04.112005    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-8sn9v"]
Jul 17 11:42:04.120832 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:04.120810    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-8sn9v"]
Jul 17 11:42:05.664910 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:05.664880    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85c3cc4d-ee09-424b-9e1d-306b8bce10a2" path="/var/lib/kubelet/pods/85c3cc4d-ee09-424b-9e1d-306b8bce10a2/volumes"
Jul 17 11:42:16.144424 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:16.144386    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:42:17.479686 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:17.479603    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:42:23.013522 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:23.013478    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:42:25.763178 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:25.763150    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nj5mj_55d1de37-8635-4de8-ab59-4ebfd0346a40/ovn-acl-logging/0.log"
Jul 17 11:42:25.770901 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:25.770878    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nj5mj_55d1de37-8635-4de8-ab59-4ebfd0346a40/ovn-acl-logging/0.log"
Jul 17 11:42:29.297434 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:29.297396    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:42:29.414302 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:29.414272    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:42:36.818708 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:36.818668    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:42:37.052273 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:37.052241    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:42:37.582322 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:37.582286    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-tnfkd"]
Jul 17 11:42:37.582646 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:37.582633    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85c3cc4d-ee09-424b-9e1d-306b8bce10a2" containerName="revoke-keys"
Jul 17 11:42:37.582694 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:37.582649    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="85c3cc4d-ee09-424b-9e1d-306b8bce10a2" containerName="revoke-keys"
Jul 17 11:42:37.582694 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:37.582657    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85c3cc4d-ee09-424b-9e1d-306b8bce10a2" containerName="revoke-keys"
Jul 17 11:42:37.582694 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:37.582663    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="85c3cc4d-ee09-424b-9e1d-306b8bce10a2" containerName="revoke-keys"
Jul 17 11:42:37.582793 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:37.582737    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="85c3cc4d-ee09-424b-9e1d-306b8bce10a2" containerName="revoke-keys"
Jul 17 11:42:37.582793 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:37.582749    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="85c3cc4d-ee09-424b-9e1d-306b8bce10a2" containerName="revoke-keys"
Jul 17 11:42:37.585678 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:37.585662    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-tnfkd"
Jul 17 11:42:37.588368 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:37.588349    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 11:42:37.588479 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:37.588370    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwv2h\""
Jul 17 11:42:37.594053 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:37.594030    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-tnfkd"]
Jul 17 11:42:37.740599 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:37.740567    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45989be5-tnfkd\" (UID: \"ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-tnfkd"
Jul 17 11:42:37.841909 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:37.841836    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45989be5-tnfkd\" (UID: \"ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-tnfkd"
Jul 17 11:42:37.842480 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:37.842460    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45989be5-tnfkd\" (UID: \"ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-tnfkd"
Jul 17 11:42:37.896604 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:37.896575    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-tnfkd"
Jul 17 11:42:38.030841 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:38.029246    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-tnfkd"]
Jul 17 11:42:38.199605 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:38.199573    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:42:38.257640 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:38.257607    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-tnfkd" event={"ID":"ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844","Type":"ContainerStarted","Data":"fe803ad9ba806ec055d60e28c20cf2c8217ddb1e51416fd047125735985838ca"}
Jul 17 11:42:38.302616 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:38.302583    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:42:38.410415 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:38.410373    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:42:39.169089 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:39.169047    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:42:39.264424 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:39.264393    2563 generic.go:356] "Generic (PLEG): container finished" podID="ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844" containerID="86c7f04fd6efbc77b0e37646f68a6ce1125c80825e7965a07379c140635c542a" exitCode=6
Jul 17 11:42:39.264571 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:39.264474    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-tnfkd" event={"ID":"ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844","Type":"ContainerDied","Data":"86c7f04fd6efbc77b0e37646f68a6ce1125c80825e7965a07379c140635c542a"}
Jul 17 11:42:39.264635 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:39.264623    2563 scope.go:117] "RemoveContainer" containerID="86c7f04fd6efbc77b0e37646f68a6ce1125c80825e7965a07379c140635c542a"
Jul 17 11:42:39.268561 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:39.268536    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:42:40.271119 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:40.271063    2563 generic.go:356] "Generic (PLEG): container finished" podID="ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844" containerID="c38771d4b7b0464a2432fc0da47f144625eee9d12300656e957054a8c636e573" exitCode=6
Jul 17 11:42:40.271119 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:40.271124    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-tnfkd" event={"ID":"ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844","Type":"ContainerDied","Data":"c38771d4b7b0464a2432fc0da47f144625eee9d12300656e957054a8c636e573"}
Jul 17 11:42:40.271646 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:40.271185    2563 scope.go:117] "RemoveContainer" containerID="86c7f04fd6efbc77b0e37646f68a6ce1125c80825e7965a07379c140635c542a"
Jul 17 11:42:40.271646 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:40.271373    2563 scope.go:117] "RemoveContainer" containerID="c38771d4b7b0464a2432fc0da47f144625eee9d12300656e957054a8c636e573"
Jul 17 11:42:40.271646 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:42:40.271608    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45989be5-tnfkd_odh-ai-gateway-infra(ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-tnfkd" podUID="ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844"
Jul 17 11:42:40.817165 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:40.817130    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:42:41.277909 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:41.277884    2563 scope.go:117] "RemoveContainer" containerID="c38771d4b7b0464a2432fc0da47f144625eee9d12300656e957054a8c636e573"
Jul 17 11:42:41.278355 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:42:41.278148    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45989be5-tnfkd_odh-ai-gateway-infra(ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-tnfkd" podUID="ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844"
Jul 17 11:42:48.170505 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:48.170464    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:42:48.264675 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:48.264644    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:42:48.854973 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:48.854936    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:42:51.660195 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:51.660160    2563 scope.go:117] "RemoveContainer" containerID="c38771d4b7b0464a2432fc0da47f144625eee9d12300656e957054a8c636e573"
Jul 17 11:42:52.331728 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:52.331700    2563 generic.go:356] "Generic (PLEG): container finished" podID="ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844" containerID="2091195a636e79fd39398f5e6f4b67f0ba5ac90c6b553e9f9f7c70437384f8ae" exitCode=6
Jul 17 11:42:52.331851 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:52.331775    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-tnfkd" event={"ID":"ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844","Type":"ContainerDied","Data":"2091195a636e79fd39398f5e6f4b67f0ba5ac90c6b553e9f9f7c70437384f8ae"}
Jul 17 11:42:52.331851 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:52.331835    2563 scope.go:117] "RemoveContainer" containerID="c38771d4b7b0464a2432fc0da47f144625eee9d12300656e957054a8c636e573"
Jul 17 11:42:52.332211 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:52.332185    2563 scope.go:117] "RemoveContainer" containerID="2091195a636e79fd39398f5e6f4b67f0ba5ac90c6b553e9f9f7c70437384f8ae"
Jul 17 11:42:52.332417 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:42:52.332389    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45989be5-tnfkd_odh-ai-gateway-infra(ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-tnfkd" podUID="ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844"
Jul 17 11:42:53.370117 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:53.370064    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-tnfkd"]
Jul 17 11:42:53.502176 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:53.502150    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-tnfkd"
Jul 17 11:42:53.573339 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:53.573316    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844-maas-api-service-ca\") pod \"ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844\" (UID: \"ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844\") "
Jul 17 11:42:53.573638 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:53.573618    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844" (UID: "ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:42:53.674012 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:53.673979    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:42:54.344843 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:54.344806    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-tnfkd" event={"ID":"ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844","Type":"ContainerDied","Data":"fe803ad9ba806ec055d60e28c20cf2c8217ddb1e51416fd047125735985838ca"}
Jul 17 11:42:54.344843 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:54.344830    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-tnfkd"
Jul 17 11:42:54.345095 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:54.344864    2563 scope.go:117] "RemoveContainer" containerID="2091195a636e79fd39398f5e6f4b67f0ba5ac90c6b553e9f9f7c70437384f8ae"
Jul 17 11:42:54.367307 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:54.367275    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-tnfkd"]
Jul 17 11:42:54.380942 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:54.378672    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-tnfkd"]
Jul 17 11:42:55.417007 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:55.416974    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:42:55.517093 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:55.517037    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:42:55.613581 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:55.613549    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:42:55.664835 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:42:55.664807    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844" path="/var/lib/kubelet/pods/ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844/volumes"
Jul 17 11:43:06.204566 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:06.204532    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:43:06.588900 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:06.588807    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:43:27.638845 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:27.638807    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:43:33.886643 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:33.886605    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:43:33.986606 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:33.986571    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:43:34.317640 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:34.317553    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:43:34.594298 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:34.594217    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:43:35.001446 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:35.001412    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:43:37.419882 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:37.419842    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:43:37.562056 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:37.562022    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:43:37.662406 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:37.662368    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:43:37.839450 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:37.839370    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:43:38.228444 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:38.228408    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:43:38.331404 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:38.331369    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:43:38.841664 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:38.841633    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs"]
Jul 17 11:43:38.842059 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:38.841947    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85c3cc4d-ee09-424b-9e1d-306b8bce10a2" containerName="revoke-keys"
Jul 17 11:43:38.842059 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:38.841976    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="85c3cc4d-ee09-424b-9e1d-306b8bce10a2" containerName="revoke-keys"
Jul 17 11:43:38.842059 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:38.841988    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844" containerName="revoke-keys"
Jul 17 11:43:38.842059 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:38.841993    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844" containerName="revoke-keys"
Jul 17 11:43:38.842059 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:38.842002    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844" containerName="revoke-keys"
Jul 17 11:43:38.842059 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:38.842008    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844" containerName="revoke-keys"
Jul 17 11:43:38.842059 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:38.842015    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844" containerName="revoke-keys"
Jul 17 11:43:38.842059 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:38.842022    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844" containerName="revoke-keys"
Jul 17 11:43:38.842330 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:38.842122    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844" containerName="revoke-keys"
Jul 17 11:43:38.842330 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:38.842134    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="85c3cc4d-ee09-424b-9e1d-306b8bce10a2" containerName="revoke-keys"
Jul 17 11:43:38.842330 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:38.842142    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844" containerName="revoke-keys"
Jul 17 11:43:38.842330 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:38.842149    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca5f8a8e-c6ad-46ef-adcc-0ad2bb336844" containerName="revoke-keys"
Jul 17 11:43:38.844856 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:38.844840    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs"
Jul 17 11:43:38.847545 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:38.847514    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 11:43:38.847665 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:38.847519    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwv2h\""
Jul 17 11:43:38.853888 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:38.853845    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs"]
Jul 17 11:43:38.859210 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:38.859184    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4caa361a-5949-409c-ae82-60bdcf3b8489-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs\" (UID: \"4caa361a-5949-409c-ae82-60bdcf3b8489\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs"
Jul 17 11:43:38.960301 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:38.960269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4caa361a-5949-409c-ae82-60bdcf3b8489-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs\" (UID: \"4caa361a-5949-409c-ae82-60bdcf3b8489\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs"
Jul 17 11:43:38.960908 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:38.960856    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4caa361a-5949-409c-ae82-60bdcf3b8489-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs\" (UID: \"4caa361a-5949-409c-ae82-60bdcf3b8489\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs"
Jul 17 11:43:39.156870 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:39.156833    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs"
Jul 17 11:43:39.289387 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:39.289353    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs"]
Jul 17 11:43:39.290612 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:43:39.290585    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4caa361a_5949_409c_ae82_60bdcf3b8489.slice/crio-680dd7700436ab75cebe9923ad6ba28cee2effd3511bbe99fc7c24b9a4a6e02e WatchSource:0}: Error finding container 680dd7700436ab75cebe9923ad6ba28cee2effd3511bbe99fc7c24b9a4a6e02e: Status 404 returned error can't find the container with id 680dd7700436ab75cebe9923ad6ba28cee2effd3511bbe99fc7c24b9a4a6e02e
Jul 17 11:43:39.562743 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:39.562704    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs" event={"ID":"4caa361a-5949-409c-ae82-60bdcf3b8489","Type":"ContainerStarted","Data":"be7bfd99de000aff72f0462e6c913e5598bb899f1ea6470865037d6b73778330"}
Jul 17 11:43:39.562963 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:39.562765    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs" event={"ID":"4caa361a-5949-409c-ae82-60bdcf3b8489","Type":"ContainerStarted","Data":"680dd7700436ab75cebe9923ad6ba28cee2effd3511bbe99fc7c24b9a4a6e02e"}
Jul 17 11:43:39.581868 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:39.581823    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs" podStartSLOduration=1.581808148 podStartE2EDuration="1.581808148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:43:39.580340571 +0000 UTC m=+2774.509880211" watchObservedRunningTime="2026-07-17 11:43:39.581808148 +0000 UTC m=+2774.511347788"
Jul 17 11:43:40.568983 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:40.568945    2563 generic.go:356] "Generic (PLEG): container finished" podID="4caa361a-5949-409c-ae82-60bdcf3b8489" containerID="be7bfd99de000aff72f0462e6c913e5598bb899f1ea6470865037d6b73778330" exitCode=6
Jul 17 11:43:40.569375 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:40.569007    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs" event={"ID":"4caa361a-5949-409c-ae82-60bdcf3b8489","Type":"ContainerDied","Data":"be7bfd99de000aff72f0462e6c913e5598bb899f1ea6470865037d6b73778330"}
Jul 17 11:43:40.569375 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:40.569306    2563 scope.go:117] "RemoveContainer" containerID="be7bfd99de000aff72f0462e6c913e5598bb899f1ea6470865037d6b73778330"
Jul 17 11:43:41.580160 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:41.580054    2563 generic.go:356] "Generic (PLEG): container finished" podID="4caa361a-5949-409c-ae82-60bdcf3b8489" containerID="280eed48b69912e3813950bb8f1ea94cc6ad53060d59b932305fc03ca545cd9c" exitCode=6
Jul 17 11:43:41.580160 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:41.580098    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs" event={"ID":"4caa361a-5949-409c-ae82-60bdcf3b8489","Type":"ContainerDied","Data":"280eed48b69912e3813950bb8f1ea94cc6ad53060d59b932305fc03ca545cd9c"}
Jul 17 11:43:41.580598 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:41.580161    2563 scope.go:117] "RemoveContainer" containerID="be7bfd99de000aff72f0462e6c913e5598bb899f1ea6470865037d6b73778330"
Jul 17 11:43:41.580598 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:41.580368    2563 scope.go:117] "RemoveContainer" containerID="280eed48b69912e3813950bb8f1ea94cc6ad53060d59b932305fc03ca545cd9c"
Jul 17 11:43:41.580598 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:43:41.580553    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs_odh-ai-gateway-infra(4caa361a-5949-409c-ae82-60bdcf3b8489)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs" podUID="4caa361a-5949-409c-ae82-60bdcf3b8489"
Jul 17 11:43:41.893410 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:41.893384    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:43:42.282590 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.282506    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:43:42.588879 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.588790    2563 scope.go:117] "RemoveContainer" containerID="280eed48b69912e3813950bb8f1ea94cc6ad53060d59b932305fc03ca545cd9c"
Jul 17 11:43:42.589309 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:43:42.589034    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs_odh-ai-gateway-infra(4caa361a-5949-409c-ae82-60bdcf3b8489)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs" podUID="4caa361a-5949-409c-ae82-60bdcf3b8489"
Jul 17 11:43:42.594460 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.594435    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"]
Jul 17 11:43:42.599478 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.599457    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.602311 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.602293    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7c5b42-openshift-default-dockercfg-5wjkn\""
Jul 17 11:43:42.610934 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.610912    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"]
Jul 17 11:43:42.628412 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.628377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:43:42.690125 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.690089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-workload-socket\") pod \"e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") " pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.690125 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.690129    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-workload-certs\") pod \"e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") " pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.690354 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.690158    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-istio-envoy\") pod \"e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") " pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.690354 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.690180    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4398a8cd-4110-4c03-99d1-1f0764e744f8-istio-podinfo\") pod \"e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") " pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.690354 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.690200    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4398a8cd-4110-4c03-99d1-1f0764e744f8-istio-token\") pod \"e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") " pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.690354 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.690293    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lczpg\" (UniqueName: \"kubernetes.io/projected/4398a8cd-4110-4c03-99d1-1f0764e744f8-kube-api-access-lczpg\") pod \"e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") " pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.690354 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.690324    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-credential-socket\") pod \"e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") " pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.690547 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.690379    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-istio-data\") pod \"e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") " pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.690547 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.690410    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4398a8cd-4110-4c03-99d1-1f0764e744f8-istiod-ca-cert\") pod \"e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") " pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.791435 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.791398    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-workload-certs\") pod \"e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") " pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.791601 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.791450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-istio-envoy\") pod \"e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") " pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.791601 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.791474    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4398a8cd-4110-4c03-99d1-1f0764e744f8-istio-podinfo\") pod \"e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") " pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.791601 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.791496    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4398a8cd-4110-4c03-99d1-1f0764e744f8-istio-token\") pod \"e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") " pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.791775 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.791653    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lczpg\" (UniqueName: \"kubernetes.io/projected/4398a8cd-4110-4c03-99d1-1f0764e744f8-kube-api-access-lczpg\") pod \"e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") " pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.791775 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.791701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-credential-socket\") pod \"e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") " pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.791775 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.791765    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-istio-data\") pod \"e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") " pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.791938 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.791809    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-workload-certs\") pod \"e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") " pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.791938 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.791814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4398a8cd-4110-4c03-99d1-1f0764e744f8-istiod-ca-cert\") pod \"e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") " pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.792035 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.791991    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-workload-socket\") pod \"e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") " pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.792106 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.792086    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-credential-socket\") pod \"e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") " pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.792236 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.792197    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-istio-data\") pod \"e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") " pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.792358 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.792340    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-workload-socket\") pod \"e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") " pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.792504 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.792484    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4398a8cd-4110-4c03-99d1-1f0764e744f8-istiod-ca-cert\") pod \"e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") " pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.793770 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.793743    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-istio-envoy\") pod \"e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") " pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.794011 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.793989    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4398a8cd-4110-4c03-99d1-1f0764e744f8-istio-podinfo\") pod \"e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") " pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.801413 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.801390    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4398a8cd-4110-4c03-99d1-1f0764e744f8-istio-token\") pod \"e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") " pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.801640 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.801616    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lczpg\" (UniqueName: \"kubernetes.io/projected/4398a8cd-4110-4c03-99d1-1f0764e744f8-kube-api-access-lczpg\") pod \"e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") " pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:42.911452 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:42.911419    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:43.065261 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:43.065228    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"]
Jul 17 11:43:43.066399 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:43:43.066372    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4398a8cd_4110_4c03_99d1_1f0764e744f8.slice/crio-2681d7ded6f62befaea9e1209c1ea85660eba4908bdcc8091ff42b854aab9162 WatchSource:0}: Error finding container 2681d7ded6f62befaea9e1209c1ea85660eba4908bdcc8091ff42b854aab9162: Status 404 returned error can't find the container with id 2681d7ded6f62befaea9e1209c1ea85660eba4908bdcc8091ff42b854aab9162
Jul 17 11:43:43.069344 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:43.069306    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 17 11:43:43.069449 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:43.069381    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 17 11:43:43.069449 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:43.069420    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 17 11:43:43.271884 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:43.271795    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:43:43.595118 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:43.595001    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh" event={"ID":"4398a8cd-4110-4c03-99d1-1f0764e744f8","Type":"ContainerStarted","Data":"3858b971a4be0ac60746768508d54174d70ec38620ac882b9f9a75d380c5eac7"}
Jul 17 11:43:43.595118 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:43.595094    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh" event={"ID":"4398a8cd-4110-4c03-99d1-1f0764e744f8","Type":"ContainerStarted","Data":"2681d7ded6f62befaea9e1209c1ea85660eba4908bdcc8091ff42b854aab9162"}
Jul 17 11:43:43.620498 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:43.620449    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh" podStartSLOduration=1.620431698 podStartE2EDuration="1.620431698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:43:43.620359701 +0000 UTC m=+2778.549899366" watchObservedRunningTime="2026-07-17 11:43:43.620431698 +0000 UTC m=+2778.549971342"
Jul 17 11:43:43.911769 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:43.911740    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:44.917279 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:44.917245    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:45.606483 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:45.606454    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:45.607358 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:45.607336    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:43:45.641602 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:45.641565    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:43:45.744162 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:45.744121    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:43:45.840134 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:45.840096    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:43:46.491791 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:46.491757    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:43:46.608867 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:46.608833    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:43:50.139970 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:50.139935    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:43:55.199473 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.199441    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"]
Jul 17 11:43:55.202591 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.202556    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"
Jul 17 11:43:55.205783 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.205761    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fc3287\"/\"default-dockercfg-tr97z\""
Jul 17 11:43:55.205898 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.205843    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fc3287\"/\"kube-root-ca.crt\""
Jul 17 11:43:55.207117 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.207100    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fc3287\"/\"openshift-service-ca.crt\""
Jul 17 11:43:55.207371 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.207144    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fc3287\"/\"test-model-fc3287-kserve-self-signed-certs\""
Jul 17 11:43:55.212557 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.212537    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"]
Jul 17 11:43:55.236450 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.236419    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:43:55.308438 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.308402    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe609732-2fdf-4fd5-aa8c-95177561f2ac-tmp-dir\") pod \"test-model-fc3287-kserve-8d674ff5-svbsp\" (UID: \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\") " pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"
Jul 17 11:43:55.308438 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.308444    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe609732-2fdf-4fd5-aa8c-95177561f2ac-dshm\") pod \"test-model-fc3287-kserve-8d674ff5-svbsp\" (UID: \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\") " pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"
Jul 17 11:43:55.308641 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.308467    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe609732-2fdf-4fd5-aa8c-95177561f2ac-tls-certs\") pod \"test-model-fc3287-kserve-8d674ff5-svbsp\" (UID: \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\") " pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"
Jul 17 11:43:55.308641 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.308484    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgmsr\" (UniqueName: \"kubernetes.io/projected/fe609732-2fdf-4fd5-aa8c-95177561f2ac-kube-api-access-wgmsr\") pod \"test-model-fc3287-kserve-8d674ff5-svbsp\" (UID: \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\") " pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"
Jul 17 11:43:55.308641 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.308535    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe609732-2fdf-4fd5-aa8c-95177561f2ac-model-cache\") pod \"test-model-fc3287-kserve-8d674ff5-svbsp\" (UID: \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\") " pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"
Jul 17 11:43:55.308641 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.308598    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe609732-2fdf-4fd5-aa8c-95177561f2ac-home\") pod \"test-model-fc3287-kserve-8d674ff5-svbsp\" (UID: \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\") " pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"
Jul 17 11:43:55.409320 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.409282    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe609732-2fdf-4fd5-aa8c-95177561f2ac-model-cache\") pod \"test-model-fc3287-kserve-8d674ff5-svbsp\" (UID: \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\") " pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"
Jul 17 11:43:55.409523 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.409383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe609732-2fdf-4fd5-aa8c-95177561f2ac-home\") pod \"test-model-fc3287-kserve-8d674ff5-svbsp\" (UID: \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\") " pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"
Jul 17 11:43:55.409523 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.409430    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe609732-2fdf-4fd5-aa8c-95177561f2ac-tmp-dir\") pod \"test-model-fc3287-kserve-8d674ff5-svbsp\" (UID: \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\") " pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"
Jul 17 11:43:55.409523 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.409466    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe609732-2fdf-4fd5-aa8c-95177561f2ac-dshm\") pod \"test-model-fc3287-kserve-8d674ff5-svbsp\" (UID: \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\") " pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"
Jul 17 11:43:55.409523 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.409495    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe609732-2fdf-4fd5-aa8c-95177561f2ac-tls-certs\") pod \"test-model-fc3287-kserve-8d674ff5-svbsp\" (UID: \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\") " pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"
Jul 17 11:43:55.409523 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.409516    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgmsr\" (UniqueName: \"kubernetes.io/projected/fe609732-2fdf-4fd5-aa8c-95177561f2ac-kube-api-access-wgmsr\") pod \"test-model-fc3287-kserve-8d674ff5-svbsp\" (UID: \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\") " pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"
Jul 17 11:43:55.409776 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.409701    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe609732-2fdf-4fd5-aa8c-95177561f2ac-model-cache\") pod \"test-model-fc3287-kserve-8d674ff5-svbsp\" (UID: \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\") " pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"
Jul 17 11:43:55.409832 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.409782    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe609732-2fdf-4fd5-aa8c-95177561f2ac-home\") pod \"test-model-fc3287-kserve-8d674ff5-svbsp\" (UID: \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\") " pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"
Jul 17 11:43:55.409832 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.409809    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe609732-2fdf-4fd5-aa8c-95177561f2ac-tmp-dir\") pod \"test-model-fc3287-kserve-8d674ff5-svbsp\" (UID: \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\") " pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"
Jul 17 11:43:55.411654 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.411636    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe609732-2fdf-4fd5-aa8c-95177561f2ac-dshm\") pod \"test-model-fc3287-kserve-8d674ff5-svbsp\" (UID: \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\") " pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"
Jul 17 11:43:55.411957 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.411942    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe609732-2fdf-4fd5-aa8c-95177561f2ac-tls-certs\") pod \"test-model-fc3287-kserve-8d674ff5-svbsp\" (UID: \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\") " pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"
Jul 17 11:43:55.419145 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.419119    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgmsr\" (UniqueName: \"kubernetes.io/projected/fe609732-2fdf-4fd5-aa8c-95177561f2ac-kube-api-access-wgmsr\") pod \"test-model-fc3287-kserve-8d674ff5-svbsp\" (UID: \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\") " pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"
Jul 17 11:43:55.512900 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.512806    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"
Jul 17 11:43:55.673528 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.673500    2563 scope.go:117] "RemoveContainer" containerID="280eed48b69912e3813950bb8f1ea94cc6ad53060d59b932305fc03ca545cd9c"
Jul 17 11:43:55.690746 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:55.690725    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"]
Jul 17 11:43:55.692149 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:43:55.692118    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe609732_2fdf_4fd5_aa8c_95177561f2ac.slice/crio-1982849d1bb378d5a1ef05039d9ffb72225957d495e6e952e2dd3a0c84a59b11 WatchSource:0}: Error finding container 1982849d1bb378d5a1ef05039d9ffb72225957d495e6e952e2dd3a0c84a59b11: Status 404 returned error can't find the container with id 1982849d1bb378d5a1ef05039d9ffb72225957d495e6e952e2dd3a0c84a59b11
Jul 17 11:43:56.683232 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:56.683196    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp" event={"ID":"fe609732-2fdf-4fd5-aa8c-95177561f2ac","Type":"ContainerStarted","Data":"c371092545393d4538bf8d8b5a0249cb8d4ac72a82e8a9bce86a825aaaa664e6"}
Jul 17 11:43:56.683673 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:56.683242    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp" event={"ID":"fe609732-2fdf-4fd5-aa8c-95177561f2ac","Type":"ContainerStarted","Data":"1982849d1bb378d5a1ef05039d9ffb72225957d495e6e952e2dd3a0c84a59b11"}
Jul 17 11:43:56.685147 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:56.685125    2563 generic.go:356] "Generic (PLEG): container finished" podID="4caa361a-5949-409c-ae82-60bdcf3b8489" containerID="7c1679e5862fcf12b2265d45c5996ad1819e25bd972c87c94bd25ce856cc12a2" exitCode=6
Jul 17 11:43:56.685259 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:56.685187    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs" event={"ID":"4caa361a-5949-409c-ae82-60bdcf3b8489","Type":"ContainerDied","Data":"7c1679e5862fcf12b2265d45c5996ad1819e25bd972c87c94bd25ce856cc12a2"}
Jul 17 11:43:56.685259 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:56.685251    2563 scope.go:117] "RemoveContainer" containerID="280eed48b69912e3813950bb8f1ea94cc6ad53060d59b932305fc03ca545cd9c"
Jul 17 11:43:56.685496 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:56.685481    2563 scope.go:117] "RemoveContainer" containerID="7c1679e5862fcf12b2265d45c5996ad1819e25bd972c87c94bd25ce856cc12a2"
Jul 17 11:43:56.685710 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:43:56.685690    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs_odh-ai-gateway-infra(4caa361a-5949-409c-ae82-60bdcf3b8489)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs" podUID="4caa361a-5949-409c-ae82-60bdcf3b8489"
Jul 17 11:43:56.706217 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:56.705624    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp" podStartSLOduration=1.705605556 podStartE2EDuration="1.705605556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:43:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:43:56.701033718 +0000 UTC m=+2791.630573368" watchObservedRunningTime="2026-07-17 11:43:56.705605556 +0000 UTC m=+2791.635145197"
Jul 17 11:43:57.733360 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:57.733319    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs"]
Jul 17 11:43:57.877291 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:57.877267    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs"
Jul 17 11:43:57.932060 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:57.932027    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4caa361a-5949-409c-ae82-60bdcf3b8489-maas-api-service-ca\") pod \"4caa361a-5949-409c-ae82-60bdcf3b8489\" (UID: \"4caa361a-5949-409c-ae82-60bdcf3b8489\") "
Jul 17 11:43:57.932396 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:57.932374    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4caa361a-5949-409c-ae82-60bdcf3b8489-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4caa361a-5949-409c-ae82-60bdcf3b8489" (UID: "4caa361a-5949-409c-ae82-60bdcf3b8489"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:43:58.033537 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:58.033440    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4caa361a-5949-409c-ae82-60bdcf3b8489-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:43:58.701448 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:58.701214    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs"
Jul 17 11:43:58.701448 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:58.701244    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs" event={"ID":"4caa361a-5949-409c-ae82-60bdcf3b8489","Type":"ContainerDied","Data":"680dd7700436ab75cebe9923ad6ba28cee2effd3511bbe99fc7c24b9a4a6e02e"}
Jul 17 11:43:58.701448 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:58.701307    2563 scope.go:117] "RemoveContainer" containerID="7c1679e5862fcf12b2265d45c5996ad1819e25bd972c87c94bd25ce856cc12a2"
Jul 17 11:43:58.743614 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:58.743587    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs"]
Jul 17 11:43:58.751672 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:58.751649    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-8c5gs"]
Jul 17 11:43:59.296400 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:59.296360    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:43:59.395927 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:59.395884    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:43:59.667014 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:43:59.666981    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4caa361a-5949-409c-ae82-60bdcf3b8489" path="/var/lib/kubelet/pods/4caa361a-5949-409c-ae82-60bdcf3b8489/volumes"
Jul 17 11:44:00.608362 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:00.608278    2563 patch_prober.go:28] interesting pod/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 17 11:44:00.608362 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:00.608329    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh" podUID="4398a8cd-4110-4c03-99d1-1f0764e744f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 17 11:44:00.712962 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:00.712930    2563 generic.go:356] "Generic (PLEG): container finished" podID="4398a8cd-4110-4c03-99d1-1f0764e744f8" containerID="3858b971a4be0ac60746768508d54174d70ec38620ac882b9f9a75d380c5eac7" exitCode=0
Jul 17 11:44:00.713140 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:00.713000    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh" event={"ID":"4398a8cd-4110-4c03-99d1-1f0764e744f8","Type":"ContainerDied","Data":"3858b971a4be0ac60746768508d54174d70ec38620ac882b9f9a75d380c5eac7"}
Jul 17 11:44:00.713434 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:00.713389    2563 scope.go:117] "RemoveContainer" containerID="3858b971a4be0ac60746768508d54174d70ec38620ac882b9f9a75d380c5eac7"
Jul 17 11:44:00.714364 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:00.714342    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 17 11:44:00.714419 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:00.714407    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 17 11:44:00.714457 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:00.714440    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 17 11:44:01.719613 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:01.719576    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh" event={"ID":"4398a8cd-4110-4c03-99d1-1f0764e744f8","Type":"ContainerStarted","Data":"d1050c69447beee4d1f333df2ba1926bff784d2ae53d294e81fff02f5f184537"}
Jul 17 11:44:01.912199 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:01.912157    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:44:01.912418 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:01.912392    2563 patch_prober.go:28] interesting pod/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 17 11:44:01.912496 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:01.912452    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh" podUID="4398a8cd-4110-4c03-99d1-1f0764e744f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 17 11:44:02.912493 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:02.912450    2563 patch_prober.go:28] interesting pod/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 17 11:44:02.912904 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:02.912509    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh" podUID="4398a8cd-4110-4c03-99d1-1f0764e744f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 17 11:44:03.912586 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:03.912546    2563 patch_prober.go:28] interesting pod/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 17 11:44:03.913053 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:03.912617    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh" podUID="4398a8cd-4110-4c03-99d1-1f0764e744f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 17 11:44:04.912305 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:04.912267    2563 patch_prober.go:28] interesting pod/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 17 11:44:04.912478 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:04.912319    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh" podUID="4398a8cd-4110-4c03-99d1-1f0764e744f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 17 11:44:05.512908 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:05.512876    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"
Jul 17 11:44:05.513302 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:05.512923    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"
Jul 17 11:44:05.524745 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:05.524721    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"
Jul 17 11:44:05.750235 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:05.750202    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"
Jul 17 11:44:05.915949 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:05.915918    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:44:05.916317 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:05.916301    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:44:05.916971 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:05.916954    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:44:07.583130 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:07.583097    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:44:07.683283 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:07.683255    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:44:15.794821 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:15.794787    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:44:24.437856 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:24.437806    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:44:59.365936 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:59.365884    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"]
Jul 17 11:44:59.394620 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:44:59.394582    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:00.488720 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:00.488677    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:01.324102 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:01.324040    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:01.429790 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:01.429756    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:11.139349 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:11.139311    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:14.386179 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:14.386117    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp" podUID="fe609732-2fdf-4fd5-aa8c-95177561f2ac" containerName="main" containerID="cri-o://c371092545393d4538bf8d8b5a0249cb8d4ac72a82e8a9bce86a825aaaa664e6" gracePeriod=45
Jul 17 11:45:14.637505 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:14.637450    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"
Jul 17 11:45:14.713438 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:14.713406    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgmsr\" (UniqueName: \"kubernetes.io/projected/fe609732-2fdf-4fd5-aa8c-95177561f2ac-kube-api-access-wgmsr\") pod \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\" (UID: \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\") "
Jul 17 11:45:14.713620 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:14.713460    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe609732-2fdf-4fd5-aa8c-95177561f2ac-model-cache\") pod \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\" (UID: \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\") "
Jul 17 11:45:14.713620 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:14.713483    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe609732-2fdf-4fd5-aa8c-95177561f2ac-tmp-dir\") pod \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\" (UID: \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\") "
Jul 17 11:45:14.713620 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:14.713506    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe609732-2fdf-4fd5-aa8c-95177561f2ac-tls-certs\") pod \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\" (UID: \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\") "
Jul 17 11:45:14.713620 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:14.713544    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe609732-2fdf-4fd5-aa8c-95177561f2ac-home\") pod \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\" (UID: \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\") "
Jul 17 11:45:14.713620 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:14.713576    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe609732-2fdf-4fd5-aa8c-95177561f2ac-dshm\") pod \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\" (UID: \"fe609732-2fdf-4fd5-aa8c-95177561f2ac\") "
Jul 17 11:45:14.713883 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:14.713789    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe609732-2fdf-4fd5-aa8c-95177561f2ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe609732-2fdf-4fd5-aa8c-95177561f2ac" (UID: "fe609732-2fdf-4fd5-aa8c-95177561f2ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:45:14.713883 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:14.713806    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe609732-2fdf-4fd5-aa8c-95177561f2ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe609732-2fdf-4fd5-aa8c-95177561f2ac" (UID: "fe609732-2fdf-4fd5-aa8c-95177561f2ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:45:14.713883 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:14.713835    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe609732-2fdf-4fd5-aa8c-95177561f2ac-home" (OuterVolumeSpecName: "home") pod "fe609732-2fdf-4fd5-aa8c-95177561f2ac" (UID: "fe609732-2fdf-4fd5-aa8c-95177561f2ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:45:14.714038 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:14.713911    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe609732-2fdf-4fd5-aa8c-95177561f2ac-model-cache\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:45:14.714038 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:14.713963    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe609732-2fdf-4fd5-aa8c-95177561f2ac-tmp-dir\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:45:14.715737 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:14.715714    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe609732-2fdf-4fd5-aa8c-95177561f2ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe609732-2fdf-4fd5-aa8c-95177561f2ac" (UID: "fe609732-2fdf-4fd5-aa8c-95177561f2ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 11:45:14.715825 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:14.715775    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe609732-2fdf-4fd5-aa8c-95177561f2ac-dshm" (OuterVolumeSpecName: "dshm") pod "fe609732-2fdf-4fd5-aa8c-95177561f2ac" (UID: "fe609732-2fdf-4fd5-aa8c-95177561f2ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:45:14.715870 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:14.715828    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe609732-2fdf-4fd5-aa8c-95177561f2ac-kube-api-access-wgmsr" (OuterVolumeSpecName: "kube-api-access-wgmsr") pod "fe609732-2fdf-4fd5-aa8c-95177561f2ac" (UID: "fe609732-2fdf-4fd5-aa8c-95177561f2ac"). InnerVolumeSpecName "kube-api-access-wgmsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 11:45:14.815078 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:14.815032    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe609732-2fdf-4fd5-aa8c-95177561f2ac-tls-certs\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:45:14.815078 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:14.815060    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe609732-2fdf-4fd5-aa8c-95177561f2ac-home\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:45:14.815078 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:14.815083    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe609732-2fdf-4fd5-aa8c-95177561f2ac-dshm\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:45:14.815331 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:14.815093    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgmsr\" (UniqueName: \"kubernetes.io/projected/fe609732-2fdf-4fd5-aa8c-95177561f2ac-kube-api-access-wgmsr\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:45:15.089864 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:15.089776    2563 generic.go:356] "Generic (PLEG): container finished" podID="fe609732-2fdf-4fd5-aa8c-95177561f2ac" containerID="c371092545393d4538bf8d8b5a0249cb8d4ac72a82e8a9bce86a825aaaa664e6" exitCode=0
Jul 17 11:45:15.089864 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:15.089817    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"
Jul 17 11:45:15.090061 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:15.089857    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp" event={"ID":"fe609732-2fdf-4fd5-aa8c-95177561f2ac","Type":"ContainerDied","Data":"c371092545393d4538bf8d8b5a0249cb8d4ac72a82e8a9bce86a825aaaa664e6"}
Jul 17 11:45:15.090061 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:15.089900    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp" event={"ID":"fe609732-2fdf-4fd5-aa8c-95177561f2ac","Type":"ContainerDied","Data":"1982849d1bb378d5a1ef05039d9ffb72225957d495e6e952e2dd3a0c84a59b11"}
Jul 17 11:45:15.090061 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:15.089916    2563 scope.go:117] "RemoveContainer" containerID="c371092545393d4538bf8d8b5a0249cb8d4ac72a82e8a9bce86a825aaaa664e6"
Jul 17 11:45:15.098924 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:15.098907    2563 scope.go:117] "RemoveContainer" containerID="c371092545393d4538bf8d8b5a0249cb8d4ac72a82e8a9bce86a825aaaa664e6"
Jul 17 11:45:15.099285 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:45:15.099266    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c371092545393d4538bf8d8b5a0249cb8d4ac72a82e8a9bce86a825aaaa664e6\": container with ID starting with c371092545393d4538bf8d8b5a0249cb8d4ac72a82e8a9bce86a825aaaa664e6 not found: ID does not exist" containerID="c371092545393d4538bf8d8b5a0249cb8d4ac72a82e8a9bce86a825aaaa664e6"
Jul 17 11:45:15.099342 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:15.099293    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c371092545393d4538bf8d8b5a0249cb8d4ac72a82e8a9bce86a825aaaa664e6"} err="failed to get container status \"c371092545393d4538bf8d8b5a0249cb8d4ac72a82e8a9bce86a825aaaa664e6\": rpc error: code = NotFound desc = could not find container \"c371092545393d4538bf8d8b5a0249cb8d4ac72a82e8a9bce86a825aaaa664e6\": container with ID starting with c371092545393d4538bf8d8b5a0249cb8d4ac72a82e8a9bce86a825aaaa664e6 not found: ID does not exist"
Jul 17 11:45:15.114994 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:15.114970    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"]
Jul 17 11:45:15.117007 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:15.116983    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-fc3287/test-model-fc3287-kserve-8d674ff5-svbsp"]
Jul 17 11:45:15.632868 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:15.632839    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:15.665694 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:15.665656    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe609732-2fdf-4fd5-aa8c-95177561f2ac" path="/var/lib/kubelet/pods/fe609732-2fdf-4fd5-aa8c-95177561f2ac/volumes"
Jul 17 11:45:16.580347 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:16.580298    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:22.411343 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:22.411302    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:22.491242 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:22.491211    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:26.424307 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:26.424276    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:26.649941 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:26.649901    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:27.481263 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:27.481228    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:27.581916 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:27.581882    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:37.477369 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:37.477336    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:41.902714 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:41.902678    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:48.433589 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:48.433534    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:48.534846 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:48.534802    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:52.618764 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:52.618730    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"]
Jul 17 11:45:52.619181 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:52.618942    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh" podUID="4398a8cd-4110-4c03-99d1-1f0764e744f8" containerName="istio-proxy" containerID="cri-o://d1050c69447beee4d1f333df2ba1926bff784d2ae53d294e81fff02f5f184537" gracePeriod=30
Jul 17 11:45:52.642324 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:52.642297    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:52.919783 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:52.919751    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:53.263681 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:53.263583    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:53.623733 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:53.623649    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:54.227718 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:54.227679    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:56.190952 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:56.190910    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:56.290632 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:56.290577    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:56.460903 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:56.460798    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:56.563331 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:56.563298    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:57.050626 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:57.050583    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:57.240883 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:57.240853    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:45:57.870693 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:57.870672    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:45:57.900019 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:57.899997    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-istio-envoy\") pod \"4398a8cd-4110-4c03-99d1-1f0764e744f8\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") "
Jul 17 11:45:57.900146 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:57.900034    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4398a8cd-4110-4c03-99d1-1f0764e744f8-istio-podinfo\") pod \"4398a8cd-4110-4c03-99d1-1f0764e744f8\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") "
Jul 17 11:45:57.900146 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:57.900058    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lczpg\" (UniqueName: \"kubernetes.io/projected/4398a8cd-4110-4c03-99d1-1f0764e744f8-kube-api-access-lczpg\") pod \"4398a8cd-4110-4c03-99d1-1f0764e744f8\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") "
Jul 17 11:45:57.900146 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:57.900116    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4398a8cd-4110-4c03-99d1-1f0764e744f8-istio-token\") pod \"4398a8cd-4110-4c03-99d1-1f0764e744f8\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") "
Jul 17 11:45:57.900297 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:57.900159    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4398a8cd-4110-4c03-99d1-1f0764e744f8-istiod-ca-cert\") pod \"4398a8cd-4110-4c03-99d1-1f0764e744f8\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") "
Jul 17 11:45:57.900297 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:57.900212    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-workload-certs\") pod \"4398a8cd-4110-4c03-99d1-1f0764e744f8\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") "
Jul 17 11:45:57.900297 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:57.900238    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-istio-data\") pod \"4398a8cd-4110-4c03-99d1-1f0764e744f8\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") "
Jul 17 11:45:57.900452 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:57.900304    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-workload-socket\") pod \"4398a8cd-4110-4c03-99d1-1f0764e744f8\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") "
Jul 17 11:45:57.900452 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:57.900329    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-credential-socket\") pod \"4398a8cd-4110-4c03-99d1-1f0764e744f8\" (UID: \"4398a8cd-4110-4c03-99d1-1f0764e744f8\") "
Jul 17 11:45:57.900976 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:57.900587    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4398a8cd-4110-4c03-99d1-1f0764e744f8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4398a8cd-4110-4c03-99d1-1f0764e744f8" (UID: "4398a8cd-4110-4c03-99d1-1f0764e744f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:45:57.900976 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:57.900706    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4398a8cd-4110-4c03-99d1-1f0764e744f8" (UID: "4398a8cd-4110-4c03-99d1-1f0764e744f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:45:57.900976 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:57.900870    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4398a8cd-4110-4c03-99d1-1f0764e744f8" (UID: "4398a8cd-4110-4c03-99d1-1f0764e744f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:45:57.900976 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:57.900881    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4398a8cd-4110-4c03-99d1-1f0764e744f8" (UID: "4398a8cd-4110-4c03-99d1-1f0764e744f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:45:57.902819 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:57.902783    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4398a8cd-4110-4c03-99d1-1f0764e744f8-istio-token" (OuterVolumeSpecName: "istio-token") pod "4398a8cd-4110-4c03-99d1-1f0764e744f8" (UID: "4398a8cd-4110-4c03-99d1-1f0764e744f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 11:45:57.903019 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:57.902975    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4398a8cd-4110-4c03-99d1-1f0764e744f8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4398a8cd-4110-4c03-99d1-1f0764e744f8" (UID: "4398a8cd-4110-4c03-99d1-1f0764e744f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 11:45:57.903527 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:57.903505    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4398a8cd-4110-4c03-99d1-1f0764e744f8" (UID: "4398a8cd-4110-4c03-99d1-1f0764e744f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:45:57.903685 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:57.903660    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-istio-data" (OuterVolumeSpecName: "istio-data") pod "4398a8cd-4110-4c03-99d1-1f0764e744f8" (UID: "4398a8cd-4110-4c03-99d1-1f0764e744f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:45:57.904141 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:57.904062    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4398a8cd-4110-4c03-99d1-1f0764e744f8-kube-api-access-lczpg" (OuterVolumeSpecName: "kube-api-access-lczpg") pod "4398a8cd-4110-4c03-99d1-1f0764e744f8" (UID: "4398a8cd-4110-4c03-99d1-1f0764e744f8"). InnerVolumeSpecName "kube-api-access-lczpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 11:45:58.001326 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.001272    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-istio-envoy\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:45:58.001326 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.001317    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4398a8cd-4110-4c03-99d1-1f0764e744f8-istio-podinfo\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:45:58.001326 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.001327    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lczpg\" (UniqueName: \"kubernetes.io/projected/4398a8cd-4110-4c03-99d1-1f0764e744f8-kube-api-access-lczpg\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:45:58.001326 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.001335    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4398a8cd-4110-4c03-99d1-1f0764e744f8-istio-token\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:45:58.001326 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.001344    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4398a8cd-4110-4c03-99d1-1f0764e744f8-istiod-ca-cert\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:45:58.001326 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.001351    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-workload-certs\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:45:58.001669 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.001359    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-istio-data\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:45:58.001669 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.001367    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-workload-socket\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:45:58.001669 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.001374    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4398a8cd-4110-4c03-99d1-1f0764e744f8-credential-socket\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:45:58.310846 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.310762    2563 generic.go:356] "Generic (PLEG): container finished" podID="4398a8cd-4110-4c03-99d1-1f0764e744f8" containerID="d1050c69447beee4d1f333df2ba1926bff784d2ae53d294e81fff02f5f184537" exitCode=0
Jul 17 11:45:58.310846 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.310804    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"
Jul 17 11:45:58.311322 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.310851    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh" event={"ID":"4398a8cd-4110-4c03-99d1-1f0764e744f8","Type":"ContainerDied","Data":"d1050c69447beee4d1f333df2ba1926bff784d2ae53d294e81fff02f5f184537"}
Jul 17 11:45:58.311322 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.310906    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh" event={"ID":"4398a8cd-4110-4c03-99d1-1f0764e744f8","Type":"ContainerDied","Data":"2681d7ded6f62befaea9e1209c1ea85660eba4908bdcc8091ff42b854aab9162"}
Jul 17 11:45:58.311322 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.310927    2563 scope.go:117] "RemoveContainer" containerID="d1050c69447beee4d1f333df2ba1926bff784d2ae53d294e81fff02f5f184537"
Jul 17 11:45:58.320127 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.320107    2563 scope.go:117] "RemoveContainer" containerID="3858b971a4be0ac60746768508d54174d70ec38620ac882b9f9a75d380c5eac7"
Jul 17 11:45:58.328023 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.328008    2563 scope.go:117] "RemoveContainer" containerID="d1050c69447beee4d1f333df2ba1926bff784d2ae53d294e81fff02f5f184537"
Jul 17 11:45:58.328288 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:45:58.328266    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1050c69447beee4d1f333df2ba1926bff784d2ae53d294e81fff02f5f184537\": container with ID starting with d1050c69447beee4d1f333df2ba1926bff784d2ae53d294e81fff02f5f184537 not found: ID does not exist" containerID="d1050c69447beee4d1f333df2ba1926bff784d2ae53d294e81fff02f5f184537"
Jul 17 11:45:58.328373 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.328296    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1050c69447beee4d1f333df2ba1926bff784d2ae53d294e81fff02f5f184537"} err="failed to get container status \"d1050c69447beee4d1f333df2ba1926bff784d2ae53d294e81fff02f5f184537\": rpc error: code = NotFound desc = could not find container \"d1050c69447beee4d1f333df2ba1926bff784d2ae53d294e81fff02f5f184537\": container with ID starting with d1050c69447beee4d1f333df2ba1926bff784d2ae53d294e81fff02f5f184537 not found: ID does not exist"
Jul 17 11:45:58.328373 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.328323    2563 scope.go:117] "RemoveContainer" containerID="3858b971a4be0ac60746768508d54174d70ec38620ac882b9f9a75d380c5eac7"
Jul 17 11:45:58.328572 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:45:58.328495    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3858b971a4be0ac60746768508d54174d70ec38620ac882b9f9a75d380c5eac7\": container with ID starting with 3858b971a4be0ac60746768508d54174d70ec38620ac882b9f9a75d380c5eac7 not found: ID does not exist" containerID="3858b971a4be0ac60746768508d54174d70ec38620ac882b9f9a75d380c5eac7"
Jul 17 11:45:58.328572 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.328519    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3858b971a4be0ac60746768508d54174d70ec38620ac882b9f9a75d380c5eac7"} err="failed to get container status \"3858b971a4be0ac60746768508d54174d70ec38620ac882b9f9a75d380c5eac7\": rpc error: code = NotFound desc = could not find container \"3858b971a4be0ac60746768508d54174d70ec38620ac882b9f9a75d380c5eac7\": container with ID starting with 3858b971a4be0ac60746768508d54174d70ec38620ac882b9f9a75d380c5eac7 not found: ID does not exist"
Jul 17 11:45:58.333248 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.333228    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"]
Jul 17 11:45:58.341103 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.341082    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7c5b42-openshift-default-766fd947df-g6tjh"]
Jul 17 11:45:58.912296 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.912262    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-bnz7r"]
Jul 17 11:45:58.912638 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.912625    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4caa361a-5949-409c-ae82-60bdcf3b8489" containerName="revoke-keys"
Jul 17 11:45:58.912684 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.912639    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4caa361a-5949-409c-ae82-60bdcf3b8489" containerName="revoke-keys"
Jul 17 11:45:58.912684 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.912649    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4398a8cd-4110-4c03-99d1-1f0764e744f8" containerName="istio-proxy"
Jul 17 11:45:58.912684 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.912655    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4398a8cd-4110-4c03-99d1-1f0764e744f8" containerName="istio-proxy"
Jul 17 11:45:58.912684 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.912668    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4caa361a-5949-409c-ae82-60bdcf3b8489" containerName="revoke-keys"
Jul 17 11:45:58.912684 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.912674    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4caa361a-5949-409c-ae82-60bdcf3b8489" containerName="revoke-keys"
Jul 17 11:45:58.912684 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.912682    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe609732-2fdf-4fd5-aa8c-95177561f2ac" containerName="main"
Jul 17 11:45:58.913157 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.912688    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe609732-2fdf-4fd5-aa8c-95177561f2ac" containerName="main"
Jul 17 11:45:58.913157 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.912736    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe609732-2fdf-4fd5-aa8c-95177561f2ac" containerName="main"
Jul 17 11:45:58.913157 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.912743    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4398a8cd-4110-4c03-99d1-1f0764e744f8" containerName="istio-proxy"
Jul 17 11:45:58.913157 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.912748    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4caa361a-5949-409c-ae82-60bdcf3b8489" containerName="revoke-keys"
Jul 17 11:45:58.913157 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.912754    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4398a8cd-4110-4c03-99d1-1f0764e744f8" containerName="istio-proxy"
Jul 17 11:45:58.913157 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.912762    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4caa361a-5949-409c-ae82-60bdcf3b8489" containerName="revoke-keys"
Jul 17 11:45:58.913157 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.912767    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4caa361a-5949-409c-ae82-60bdcf3b8489" containerName="revoke-keys"
Jul 17 11:45:58.915565 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.915548    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-bnz7r"
Jul 17 11:45:58.918117 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.918094    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 11:45:58.918117 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.918091    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwv2h\""
Jul 17 11:45:58.924605 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:58.924577    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-bnz7r"]
Jul 17 11:45:59.010922 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:59.010888    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b6c18e-3d25-4a5b-9f33-4c19984da6cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-40b693f1-bnz7r\" (UID: \"18b6c18e-3d25-4a5b-9f33-4c19984da6cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-bnz7r"
Jul 17 11:45:59.112269 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:59.112233    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b6c18e-3d25-4a5b-9f33-4c19984da6cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-40b693f1-bnz7r\" (UID: \"18b6c18e-3d25-4a5b-9f33-4c19984da6cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-bnz7r"
Jul 17 11:45:59.112851 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:59.112830    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b6c18e-3d25-4a5b-9f33-4c19984da6cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-40b693f1-bnz7r\" (UID: \"18b6c18e-3d25-4a5b-9f33-4c19984da6cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-bnz7r"
Jul 17 11:45:59.226566 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:59.226476    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-bnz7r"
Jul 17 11:45:59.364895 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:59.364872    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-bnz7r"]
Jul 17 11:45:59.366914 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:45:59.366886    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b6c18e_3d25_4a5b_9f33_4c19984da6cc.slice/crio-8a607df44a80e60bf27a8571566eeaacdddb99779d220de3f476981f1d5eef79 WatchSource:0}: Error finding container 8a607df44a80e60bf27a8571566eeaacdddb99779d220de3f476981f1d5eef79: Status 404 returned error can't find the container with id 8a607df44a80e60bf27a8571566eeaacdddb99779d220de3f476981f1d5eef79
Jul 17 11:45:59.665103 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:45:59.665056    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4398a8cd-4110-4c03-99d1-1f0764e744f8" path="/var/lib/kubelet/pods/4398a8cd-4110-4c03-99d1-1f0764e744f8/volumes"
Jul 17 11:46:00.324445 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:00.324348    2563 generic.go:356] "Generic (PLEG): container finished" podID="18b6c18e-3d25-4a5b-9f33-4c19984da6cc" containerID="672fadba3c4d116922aabf043dde3d819e1af01e293221a02063a2c974383221" exitCode=6
Jul 17 11:46:00.324445 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:00.324394    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-bnz7r" event={"ID":"18b6c18e-3d25-4a5b-9f33-4c19984da6cc","Type":"ContainerDied","Data":"672fadba3c4d116922aabf043dde3d819e1af01e293221a02063a2c974383221"}
Jul 17 11:46:00.324685 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:00.324465    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-bnz7r" event={"ID":"18b6c18e-3d25-4a5b-9f33-4c19984da6cc","Type":"ContainerStarted","Data":"8a607df44a80e60bf27a8571566eeaacdddb99779d220de3f476981f1d5eef79"}
Jul 17 11:46:00.324741 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:00.324697    2563 scope.go:117] "RemoveContainer" containerID="672fadba3c4d116922aabf043dde3d819e1af01e293221a02063a2c974383221"
Jul 17 11:46:00.720212 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:00.720182    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:01.203437 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:01.203357    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:01.331926 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:01.331890    2563 generic.go:356] "Generic (PLEG): container finished" podID="18b6c18e-3d25-4a5b-9f33-4c19984da6cc" containerID="aa35562b1bd8d9d949a347ff8ddf28c21ca90c9e7096254887fe3e4d3a8bd97e" exitCode=6
Jul 17 11:46:01.332094 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:01.331968    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-bnz7r" event={"ID":"18b6c18e-3d25-4a5b-9f33-4c19984da6cc","Type":"ContainerDied","Data":"aa35562b1bd8d9d949a347ff8ddf28c21ca90c9e7096254887fe3e4d3a8bd97e"}
Jul 17 11:46:01.332094 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:01.332024    2563 scope.go:117] "RemoveContainer" containerID="672fadba3c4d116922aabf043dde3d819e1af01e293221a02063a2c974383221"
Jul 17 11:46:01.332263 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:01.332248    2563 scope.go:117] "RemoveContainer" containerID="aa35562b1bd8d9d949a347ff8ddf28c21ca90c9e7096254887fe3e4d3a8bd97e"
Jul 17 11:46:01.332459 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:46:01.332439    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-40b693f1-bnz7r_odh-ai-gateway-infra(18b6c18e-3d25-4a5b-9f33-4c19984da6cc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-bnz7r" podUID="18b6c18e-3d25-4a5b-9f33-4c19984da6cc"
Jul 17 11:46:01.537058 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:01.536958    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:01.635215 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:01.635182    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:02.338549 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:02.338521    2563 scope.go:117] "RemoveContainer" containerID="aa35562b1bd8d9d949a347ff8ddf28c21ca90c9e7096254887fe3e4d3a8bd97e"
Jul 17 11:46:02.339020 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:46:02.338725    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-40b693f1-bnz7r_odh-ai-gateway-infra(18b6c18e-3d25-4a5b-9f33-4c19984da6cc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-bnz7r" podUID="18b6c18e-3d25-4a5b-9f33-4c19984da6cc"
Jul 17 11:46:04.278319 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:04.278279    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:04.374917 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:04.374878    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:04.458955 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:04.458910    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:04.589707 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:04.589622    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:04.971117 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:04.971062    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:05.071918 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:05.071886    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:08.793834 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:08.793797    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:14.660576 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:14.660541    2563 scope.go:117] "RemoveContainer" containerID="aa35562b1bd8d9d949a347ff8ddf28c21ca90c9e7096254887fe3e4d3a8bd97e"
Jul 17 11:46:15.405171 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:15.405135    2563 generic.go:356] "Generic (PLEG): container finished" podID="18b6c18e-3d25-4a5b-9f33-4c19984da6cc" containerID="908d8abc87648763ca34b8a06d359e871cc9b764471291a3fdb1de33b23be869" exitCode=6
Jul 17 11:46:15.405335 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:15.405184    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-bnz7r" event={"ID":"18b6c18e-3d25-4a5b-9f33-4c19984da6cc","Type":"ContainerDied","Data":"908d8abc87648763ca34b8a06d359e871cc9b764471291a3fdb1de33b23be869"}
Jul 17 11:46:15.405335 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:15.405243    2563 scope.go:117] "RemoveContainer" containerID="aa35562b1bd8d9d949a347ff8ddf28c21ca90c9e7096254887fe3e4d3a8bd97e"
Jul 17 11:46:15.405494 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:15.405476    2563 scope.go:117] "RemoveContainer" containerID="908d8abc87648763ca34b8a06d359e871cc9b764471291a3fdb1de33b23be869"
Jul 17 11:46:15.405696 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:46:15.405675    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-40b693f1-bnz7r_odh-ai-gateway-infra(18b6c18e-3d25-4a5b-9f33-4c19984da6cc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-bnz7r" podUID="18b6c18e-3d25-4a5b-9f33-4c19984da6cc"
Jul 17 11:46:16.434450 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:16.434417    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-bnz7r"]
Jul 17 11:46:16.568618 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:16.568590    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-bnz7r"
Jul 17 11:46:16.670852 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:16.670819    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b6c18e-3d25-4a5b-9f33-4c19984da6cc-maas-api-service-ca\") pod \"18b6c18e-3d25-4a5b-9f33-4c19984da6cc\" (UID: \"18b6c18e-3d25-4a5b-9f33-4c19984da6cc\") "
Jul 17 11:46:16.671189 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:16.671168    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18b6c18e-3d25-4a5b-9f33-4c19984da6cc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "18b6c18e-3d25-4a5b-9f33-4c19984da6cc" (UID: "18b6c18e-3d25-4a5b-9f33-4c19984da6cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:46:16.771885 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:16.771797    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b6c18e-3d25-4a5b-9f33-4c19984da6cc-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:46:17.003653 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.003618    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0bf60-mfb52"]
Jul 17 11:46:17.003973 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.003961    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4caa361a-5949-409c-ae82-60bdcf3b8489" containerName="revoke-keys"
Jul 17 11:46:17.004020 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.003974    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4caa361a-5949-409c-ae82-60bdcf3b8489" containerName="revoke-keys"
Jul 17 11:46:17.004020 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.003985    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4398a8cd-4110-4c03-99d1-1f0764e744f8" containerName="istio-proxy"
Jul 17 11:46:17.004020 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.003991    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4398a8cd-4110-4c03-99d1-1f0764e744f8" containerName="istio-proxy"
Jul 17 11:46:17.004020 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.003999    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18b6c18e-3d25-4a5b-9f33-4c19984da6cc" containerName="revoke-keys"
Jul 17 11:46:17.004020 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.004004    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="18b6c18e-3d25-4a5b-9f33-4c19984da6cc" containerName="revoke-keys"
Jul 17 11:46:17.004020 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.004011    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18b6c18e-3d25-4a5b-9f33-4c19984da6cc" containerName="revoke-keys"
Jul 17 11:46:17.004020 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.004016    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="18b6c18e-3d25-4a5b-9f33-4c19984da6cc" containerName="revoke-keys"
Jul 17 11:46:17.004257 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.004086    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="18b6c18e-3d25-4a5b-9f33-4c19984da6cc" containerName="revoke-keys"
Jul 17 11:46:17.004257 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.004094    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="18b6c18e-3d25-4a5b-9f33-4c19984da6cc" containerName="revoke-keys"
Jul 17 11:46:17.006174 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.006159    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0bf60-mfb52"
Jul 17 11:46:17.016945 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.016923    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0bf60-mfb52"]
Jul 17 11:46:17.074494 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.074408    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd7a63a9-2208-46a6-99d6-1251e3a4ad17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c0bf60-mfb52\" (UID: \"bd7a63a9-2208-46a6-99d6-1251e3a4ad17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0bf60-mfb52"
Jul 17 11:46:17.175342 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.175310    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd7a63a9-2208-46a6-99d6-1251e3a4ad17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c0bf60-mfb52\" (UID: \"bd7a63a9-2208-46a6-99d6-1251e3a4ad17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0bf60-mfb52"
Jul 17 11:46:17.175888 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.175870    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd7a63a9-2208-46a6-99d6-1251e3a4ad17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c0bf60-mfb52\" (UID: \"bd7a63a9-2208-46a6-99d6-1251e3a4ad17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0bf60-mfb52"
Jul 17 11:46:17.316299 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.316265    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0bf60-mfb52"
Jul 17 11:46:17.420562 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.420530    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-bnz7r" event={"ID":"18b6c18e-3d25-4a5b-9f33-4c19984da6cc","Type":"ContainerDied","Data":"8a607df44a80e60bf27a8571566eeaacdddb99779d220de3f476981f1d5eef79"}
Jul 17 11:46:17.420562 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.420552    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-bnz7r"
Jul 17 11:46:17.420792 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.420602    2563 scope.go:117] "RemoveContainer" containerID="908d8abc87648763ca34b8a06d359e871cc9b764471291a3fdb1de33b23be869"
Jul 17 11:46:17.446297 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.446271    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-bnz7r"]
Jul 17 11:46:17.449917 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.449894    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0bf60-mfb52"]
Jul 17 11:46:17.451965 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:46:17.451912    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7a63a9_2208_46a6_99d6_1251e3a4ad17.slice/crio-0f5c0d5c84f31c21f37b605199528e6fc6df3bdf2f5e0576b7aac998852f0861 WatchSource:0}: Error finding container 0f5c0d5c84f31c21f37b605199528e6fc6df3bdf2f5e0576b7aac998852f0861: Status 404 returned error can't find the container with id 0f5c0d5c84f31c21f37b605199528e6fc6df3bdf2f5e0576b7aac998852f0861
Jul 17 11:46:17.454585 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.454561    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-bnz7r"]
Jul 17 11:46:17.620763 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.620675    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:17.665315 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.665281    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18b6c18e-3d25-4a5b-9f33-4c19984da6cc" path="/var/lib/kubelet/pods/18b6c18e-3d25-4a5b-9f33-4c19984da6cc/volumes"
Jul 17 11:46:17.721548 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.721516    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:17.955535 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.955503    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-m6vzq"]
Jul 17 11:46:17.955847 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.955836    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18b6c18e-3d25-4a5b-9f33-4c19984da6cc" containerName="revoke-keys"
Jul 17 11:46:17.955890 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.955849    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="18b6c18e-3d25-4a5b-9f33-4c19984da6cc" containerName="revoke-keys"
Jul 17 11:46:17.955924 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.955919    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="18b6c18e-3d25-4a5b-9f33-4c19984da6cc" containerName="revoke-keys"
Jul 17 11:46:17.958168 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.958150    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-m6vzq"
Jul 17 11:46:17.967584 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:17.967558    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-m6vzq"]
Jul 17 11:46:18.083871 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:18.083815    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b17975d-97bf-4063-8f3d-6cbe41a0b476-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45989be5-m6vzq\" (UID: \"1b17975d-97bf-4063-8f3d-6cbe41a0b476\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-m6vzq"
Jul 17 11:46:18.184737 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:18.184705    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b17975d-97bf-4063-8f3d-6cbe41a0b476-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45989be5-m6vzq\" (UID: \"1b17975d-97bf-4063-8f3d-6cbe41a0b476\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-m6vzq"
Jul 17 11:46:18.185365 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:18.185345    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b17975d-97bf-4063-8f3d-6cbe41a0b476-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45989be5-m6vzq\" (UID: \"1b17975d-97bf-4063-8f3d-6cbe41a0b476\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-m6vzq"
Jul 17 11:46:18.269180 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:18.269054    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-m6vzq"
Jul 17 11:46:18.402542 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:18.402501    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-m6vzq"]
Jul 17 11:46:18.403824 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:46:18.403796    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b17975d_97bf_4063_8f3d_6cbe41a0b476.slice/crio-861ba1ac5ec8048d4219faa2604d306993027a1d4c8e402a2d2026a0acdacdbc WatchSource:0}: Error finding container 861ba1ac5ec8048d4219faa2604d306993027a1d4c8e402a2d2026a0acdacdbc: Status 404 returned error can't find the container with id 861ba1ac5ec8048d4219faa2604d306993027a1d4c8e402a2d2026a0acdacdbc
Jul 17 11:46:18.427576 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:18.427551    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-m6vzq" event={"ID":"1b17975d-97bf-4063-8f3d-6cbe41a0b476","Type":"ContainerStarted","Data":"861ba1ac5ec8048d4219faa2604d306993027a1d4c8e402a2d2026a0acdacdbc"}
Jul 17 11:46:18.429445 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:18.429420    2563 generic.go:356] "Generic (PLEG): container finished" podID="bd7a63a9-2208-46a6-99d6-1251e3a4ad17" containerID="58b5d0c57892f57b683b8d0a48089328bb2aac884325163c818b07c4ba9c95b7" exitCode=0
Jul 17 11:46:18.429531 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:18.429465    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0bf60-mfb52" event={"ID":"bd7a63a9-2208-46a6-99d6-1251e3a4ad17","Type":"ContainerDied","Data":"58b5d0c57892f57b683b8d0a48089328bb2aac884325163c818b07c4ba9c95b7"}
Jul 17 11:46:18.429531 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:18.429504    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0bf60-mfb52" event={"ID":"bd7a63a9-2208-46a6-99d6-1251e3a4ad17","Type":"ContainerStarted","Data":"0f5c0d5c84f31c21f37b605199528e6fc6df3bdf2f5e0576b7aac998852f0861"}
Jul 17 11:46:19.031087 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:19.030990    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-j8m8k"]
Jul 17 11:46:19.033384 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:19.033369    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-j8m8k"
Jul 17 11:46:19.043668 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:19.043640    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-j8m8k"]
Jul 17 11:46:19.093056 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:19.093026    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/573cbab5-8966-49fc-8b51-742a127177f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-829f05b9-j8m8k\" (UID: \"573cbab5-8966-49fc-8b51-742a127177f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-j8m8k"
Jul 17 11:46:19.193790 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:19.193752    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/573cbab5-8966-49fc-8b51-742a127177f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-829f05b9-j8m8k\" (UID: \"573cbab5-8966-49fc-8b51-742a127177f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-j8m8k"
Jul 17 11:46:19.194432 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:19.194414    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/573cbab5-8966-49fc-8b51-742a127177f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-829f05b9-j8m8k\" (UID: \"573cbab5-8966-49fc-8b51-742a127177f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-j8m8k"
Jul 17 11:46:19.344539 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:19.344450    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-j8m8k"
Jul 17 11:46:19.453987 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:19.453417    2563 scope.go:117] "RemoveContainer" containerID="b9f512ba10844cf34c0525c17567eed8ebcbaa4baf09efc8f866296aa7248298"
Jul 17 11:46:19.454279 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:19.454194    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-m6vzq" event={"ID":"1b17975d-97bf-4063-8f3d-6cbe41a0b476","Type":"ContainerDied","Data":"b9f512ba10844cf34c0525c17567eed8ebcbaa4baf09efc8f866296aa7248298"}
Jul 17 11:46:19.455469 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:19.455341    2563 generic.go:356] "Generic (PLEG): container finished" podID="1b17975d-97bf-4063-8f3d-6cbe41a0b476" containerID="b9f512ba10844cf34c0525c17567eed8ebcbaa4baf09efc8f866296aa7248298" exitCode=6
Jul 17 11:46:19.509410 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:19.509387    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-j8m8k"]
Jul 17 11:46:19.511541 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:46:19.511512    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod573cbab5_8966_49fc_8b51_742a127177f7.slice/crio-53c73f59768ad38fefb60d316d09d7b4c2b7c77d26de4d7cdd1e2b7e9d17be51 WatchSource:0}: Error finding container 53c73f59768ad38fefb60d316d09d7b4c2b7c77d26de4d7cdd1e2b7e9d17be51: Status 404 returned error can't find the container with id 53c73f59768ad38fefb60d316d09d7b4c2b7c77d26de4d7cdd1e2b7e9d17be51
Jul 17 11:46:19.599285 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:19.599237    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0bf60-mfb52"
Jul 17 11:46:19.697385 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:19.697362    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd7a63a9-2208-46a6-99d6-1251e3a4ad17-maas-api-service-ca\") pod \"bd7a63a9-2208-46a6-99d6-1251e3a4ad17\" (UID: \"bd7a63a9-2208-46a6-99d6-1251e3a4ad17\") "
Jul 17 11:46:19.697697 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:19.697677    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd7a63a9-2208-46a6-99d6-1251e3a4ad17-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bd7a63a9-2208-46a6-99d6-1251e3a4ad17" (UID: "bd7a63a9-2208-46a6-99d6-1251e3a4ad17"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:46:19.798086 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:19.798048    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd7a63a9-2208-46a6-99d6-1251e3a4ad17-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:46:20.462262 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:20.462173    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0bf60-mfb52"
Jul 17 11:46:20.462262 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:20.462172    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c0bf60-mfb52" event={"ID":"bd7a63a9-2208-46a6-99d6-1251e3a4ad17","Type":"ContainerDied","Data":"0f5c0d5c84f31c21f37b605199528e6fc6df3bdf2f5e0576b7aac998852f0861"}
Jul 17 11:46:20.462764 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:20.462301    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f5c0d5c84f31c21f37b605199528e6fc6df3bdf2f5e0576b7aac998852f0861"
Jul 17 11:46:20.463723 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:20.463702    2563 generic.go:356] "Generic (PLEG): container finished" podID="573cbab5-8966-49fc-8b51-742a127177f7" containerID="6a1e60f0e8752ed73832245d73f956b37e4f0fd13a850398e799b1f5e94b896b" exitCode=6
Jul 17 11:46:20.463868 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:20.463791    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-j8m8k" event={"ID":"573cbab5-8966-49fc-8b51-742a127177f7","Type":"ContainerDied","Data":"6a1e60f0e8752ed73832245d73f956b37e4f0fd13a850398e799b1f5e94b896b"}
Jul 17 11:46:20.463868 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:20.463837    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-j8m8k" event={"ID":"573cbab5-8966-49fc-8b51-742a127177f7","Type":"ContainerStarted","Data":"53c73f59768ad38fefb60d316d09d7b4c2b7c77d26de4d7cdd1e2b7e9d17be51"}
Jul 17 11:46:20.463977 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:20.463961    2563 scope.go:117] "RemoveContainer" containerID="6a1e60f0e8752ed73832245d73f956b37e4f0fd13a850398e799b1f5e94b896b"
Jul 17 11:46:20.465839 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:20.465816    2563 generic.go:356] "Generic (PLEG): container finished" podID="1b17975d-97bf-4063-8f3d-6cbe41a0b476" containerID="71f31bf98b74fe8f33084b0d74cbcdcd0844654a3897ffd36ab3211f5f895933" exitCode=6
Jul 17 11:46:20.465966 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:20.465889    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-m6vzq" event={"ID":"1b17975d-97bf-4063-8f3d-6cbe41a0b476","Type":"ContainerDied","Data":"71f31bf98b74fe8f33084b0d74cbcdcd0844654a3897ffd36ab3211f5f895933"}
Jul 17 11:46:20.465966 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:20.465958    2563 scope.go:117] "RemoveContainer" containerID="b9f512ba10844cf34c0525c17567eed8ebcbaa4baf09efc8f866296aa7248298"
Jul 17 11:46:20.466141 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:20.466091    2563 scope.go:117] "RemoveContainer" containerID="71f31bf98b74fe8f33084b0d74cbcdcd0844654a3897ffd36ab3211f5f895933"
Jul 17 11:46:20.466326 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:46:20.466288    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45989be5-m6vzq_odh-ai-gateway-infra(1b17975d-97bf-4063-8f3d-6cbe41a0b476)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-m6vzq" podUID="1b17975d-97bf-4063-8f3d-6cbe41a0b476"
Jul 17 11:46:21.473170 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:21.473136    2563 generic.go:356] "Generic (PLEG): container finished" podID="573cbab5-8966-49fc-8b51-742a127177f7" containerID="1b07f6c3cfc7c57a8cbfdc6bd7b17030de39b31d113e3ea81e03a23428adce75" exitCode=6
Jul 17 11:46:21.473645 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:21.473222    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-j8m8k" event={"ID":"573cbab5-8966-49fc-8b51-742a127177f7","Type":"ContainerDied","Data":"1b07f6c3cfc7c57a8cbfdc6bd7b17030de39b31d113e3ea81e03a23428adce75"}
Jul 17 11:46:21.473645 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:21.473283    2563 scope.go:117] "RemoveContainer" containerID="6a1e60f0e8752ed73832245d73f956b37e4f0fd13a850398e799b1f5e94b896b"
Jul 17 11:46:21.473645 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:21.473471    2563 scope.go:117] "RemoveContainer" containerID="1b07f6c3cfc7c57a8cbfdc6bd7b17030de39b31d113e3ea81e03a23428adce75"
Jul 17 11:46:21.473809 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:46:21.473697    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-829f05b9-j8m8k_odh-ai-gateway-infra(573cbab5-8966-49fc-8b51-742a127177f7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-j8m8k" podUID="573cbab5-8966-49fc-8b51-742a127177f7"
Jul 17 11:46:21.475173 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:21.475150    2563 scope.go:117] "RemoveContainer" containerID="71f31bf98b74fe8f33084b0d74cbcdcd0844654a3897ffd36ab3211f5f895933"
Jul 17 11:46:21.475340 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:46:21.475319    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45989be5-m6vzq_odh-ai-gateway-infra(1b17975d-97bf-4063-8f3d-6cbe41a0b476)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-m6vzq" podUID="1b17975d-97bf-4063-8f3d-6cbe41a0b476"
Jul 17 11:46:22.481957 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:22.481920    2563 scope.go:117] "RemoveContainer" containerID="1b07f6c3cfc7c57a8cbfdc6bd7b17030de39b31d113e3ea81e03a23428adce75"
Jul 17 11:46:22.482449 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:46:22.482182    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-829f05b9-j8m8k_odh-ai-gateway-infra(573cbab5-8966-49fc-8b51-742a127177f7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-j8m8k" podUID="573cbab5-8966-49fc-8b51-742a127177f7"
Jul 17 11:46:25.870335 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:25.870299    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:25.969958 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:25.969922    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:27.625838 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:27.625807    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:27.726469 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:27.726430    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:27.923996 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:27.923949    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:28.672439 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:28.672401    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:32.118650 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:32.118610    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:32.660555 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:32.660525    2563 scope.go:117] "RemoveContainer" containerID="71f31bf98b74fe8f33084b0d74cbcdcd0844654a3897ffd36ab3211f5f895933"
Jul 17 11:46:33.536639 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:33.536565    2563 generic.go:356] "Generic (PLEG): container finished" podID="1b17975d-97bf-4063-8f3d-6cbe41a0b476" containerID="e0acc541779634d8dbe2d22b7b0e906f8491b2a32c67d1683da7a898bffd7f35" exitCode=6
Jul 17 11:46:33.537057 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:33.536635    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-m6vzq" event={"ID":"1b17975d-97bf-4063-8f3d-6cbe41a0b476","Type":"ContainerDied","Data":"e0acc541779634d8dbe2d22b7b0e906f8491b2a32c67d1683da7a898bffd7f35"}
Jul 17 11:46:33.537057 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:33.536710    2563 scope.go:117] "RemoveContainer" containerID="71f31bf98b74fe8f33084b0d74cbcdcd0844654a3897ffd36ab3211f5f895933"
Jul 17 11:46:33.537057 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:33.537036    2563 scope.go:117] "RemoveContainer" containerID="e0acc541779634d8dbe2d22b7b0e906f8491b2a32c67d1683da7a898bffd7f35"
Jul 17 11:46:33.537278 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:46:33.537258    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45989be5-m6vzq_odh-ai-gateway-infra(1b17975d-97bf-4063-8f3d-6cbe41a0b476)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-m6vzq" podUID="1b17975d-97bf-4063-8f3d-6cbe41a0b476"
Jul 17 11:46:34.564417 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:34.564381    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-m6vzq"]
Jul 17 11:46:34.698172 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:34.698150    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-m6vzq"
Jul 17 11:46:34.830584 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:34.830500    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b17975d-97bf-4063-8f3d-6cbe41a0b476-maas-api-service-ca\") pod \"1b17975d-97bf-4063-8f3d-6cbe41a0b476\" (UID: \"1b17975d-97bf-4063-8f3d-6cbe41a0b476\") "
Jul 17 11:46:34.830871 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:34.830851    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b17975d-97bf-4063-8f3d-6cbe41a0b476-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1b17975d-97bf-4063-8f3d-6cbe41a0b476" (UID: "1b17975d-97bf-4063-8f3d-6cbe41a0b476"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:46:34.931728 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:34.931694    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b17975d-97bf-4063-8f3d-6cbe41a0b476-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:46:35.551167 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:35.551139    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-m6vzq"
Jul 17 11:46:35.551167 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:35.551157    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-m6vzq" event={"ID":"1b17975d-97bf-4063-8f3d-6cbe41a0b476","Type":"ContainerDied","Data":"861ba1ac5ec8048d4219faa2604d306993027a1d4c8e402a2d2026a0acdacdbc"}
Jul 17 11:46:35.551426 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:35.551231    2563 scope.go:117] "RemoveContainer" containerID="e0acc541779634d8dbe2d22b7b0e906f8491b2a32c67d1683da7a898bffd7f35"
Jul 17 11:46:35.575109 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:35.575086    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-m6vzq"]
Jul 17 11:46:35.581153 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:35.581127    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-m6vzq"]
Jul 17 11:46:35.664648 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:35.664621    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b17975d-97bf-4063-8f3d-6cbe41a0b476" path="/var/lib/kubelet/pods/1b17975d-97bf-4063-8f3d-6cbe41a0b476/volumes"
Jul 17 11:46:37.660644 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:37.660610    2563 scope.go:117] "RemoveContainer" containerID="1b07f6c3cfc7c57a8cbfdc6bd7b17030de39b31d113e3ea81e03a23428adce75"
Jul 17 11:46:38.201868 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:38.201831    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:38.303208 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:38.303167    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:38.570787 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:38.570703    2563 generic.go:356] "Generic (PLEG): container finished" podID="573cbab5-8966-49fc-8b51-742a127177f7" containerID="667338945d6f6549725f99def2db04eb9387a38a35b3e5dadf7da07b98f71a71" exitCode=6
Jul 17 11:46:38.570939 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:38.570782    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-j8m8k" event={"ID":"573cbab5-8966-49fc-8b51-742a127177f7","Type":"ContainerDied","Data":"667338945d6f6549725f99def2db04eb9387a38a35b3e5dadf7da07b98f71a71"}
Jul 17 11:46:38.570939 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:38.570844    2563 scope.go:117] "RemoveContainer" containerID="1b07f6c3cfc7c57a8cbfdc6bd7b17030de39b31d113e3ea81e03a23428adce75"
Jul 17 11:46:38.571167 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:38.571151    2563 scope.go:117] "RemoveContainer" containerID="667338945d6f6549725f99def2db04eb9387a38a35b3e5dadf7da07b98f71a71"
Jul 17 11:46:38.571399 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:46:38.571359    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-829f05b9-j8m8k_odh-ai-gateway-infra(573cbab5-8966-49fc-8b51-742a127177f7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-j8m8k" podUID="573cbab5-8966-49fc-8b51-742a127177f7"
Jul 17 11:46:39.601670 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:39.601640    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-j8m8k"]
Jul 17 11:46:39.746654 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:39.746627    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-j8m8k"
Jul 17 11:46:39.877052 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:39.876965    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/573cbab5-8966-49fc-8b51-742a127177f7-maas-api-service-ca\") pod \"573cbab5-8966-49fc-8b51-742a127177f7\" (UID: \"573cbab5-8966-49fc-8b51-742a127177f7\") "
Jul 17 11:46:39.877318 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:39.877295    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/573cbab5-8966-49fc-8b51-742a127177f7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "573cbab5-8966-49fc-8b51-742a127177f7" (UID: "573cbab5-8966-49fc-8b51-742a127177f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:46:39.978136 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:39.978103    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/573cbab5-8966-49fc-8b51-742a127177f7-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:46:40.584763 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:40.584726    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-j8m8k" event={"ID":"573cbab5-8966-49fc-8b51-742a127177f7","Type":"ContainerDied","Data":"53c73f59768ad38fefb60d316d09d7b4c2b7c77d26de4d7cdd1e2b7e9d17be51"}
Jul 17 11:46:40.584971 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:40.584790    2563 scope.go:117] "RemoveContainer" containerID="667338945d6f6549725f99def2db04eb9387a38a35b3e5dadf7da07b98f71a71"
Jul 17 11:46:40.584971 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:40.584731    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-j8m8k"
Jul 17 11:46:40.607973 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:40.607949    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-j8m8k"]
Jul 17 11:46:40.613933 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:40.613904    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-829f05b9-j8m8k"]
Jul 17 11:46:41.665634 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:41.665604    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="573cbab5-8966-49fc-8b51-742a127177f7" path="/var/lib/kubelet/pods/573cbab5-8966-49fc-8b51-742a127177f7/volumes"
Jul 17 11:46:42.844971 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:42.844943    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:43.092677 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.092642    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:43.218161 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.218127    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a50d5b-gxsxr"]
Jul 17 11:46:43.218453 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.218441    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd7a63a9-2208-46a6-99d6-1251e3a4ad17" containerName="revoke-keys"
Jul 17 11:46:43.218547 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.218454    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd7a63a9-2208-46a6-99d6-1251e3a4ad17" containerName="revoke-keys"
Jul 17 11:46:43.218547 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.218470    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b17975d-97bf-4063-8f3d-6cbe41a0b476" containerName="revoke-keys"
Jul 17 11:46:43.218547 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.218476    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b17975d-97bf-4063-8f3d-6cbe41a0b476" containerName="revoke-keys"
Jul 17 11:46:43.218547 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.218485    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b17975d-97bf-4063-8f3d-6cbe41a0b476" containerName="revoke-keys"
Jul 17 11:46:43.218547 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.218490    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b17975d-97bf-4063-8f3d-6cbe41a0b476" containerName="revoke-keys"
Jul 17 11:46:43.218547 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.218497    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b17975d-97bf-4063-8f3d-6cbe41a0b476" containerName="revoke-keys"
Jul 17 11:46:43.218547 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.218502    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b17975d-97bf-4063-8f3d-6cbe41a0b476" containerName="revoke-keys"
Jul 17 11:46:43.218547 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.218513    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="573cbab5-8966-49fc-8b51-742a127177f7" containerName="revoke-keys"
Jul 17 11:46:43.218547 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.218519    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="573cbab5-8966-49fc-8b51-742a127177f7" containerName="revoke-keys"
Jul 17 11:46:43.218547 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.218525    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="573cbab5-8966-49fc-8b51-742a127177f7" containerName="revoke-keys"
Jul 17 11:46:43.218547 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.218530    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="573cbab5-8966-49fc-8b51-742a127177f7" containerName="revoke-keys"
Jul 17 11:46:43.218878 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.218578    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b17975d-97bf-4063-8f3d-6cbe41a0b476" containerName="revoke-keys"
Jul 17 11:46:43.218878 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.218586    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="573cbab5-8966-49fc-8b51-742a127177f7" containerName="revoke-keys"
Jul 17 11:46:43.218878 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.218592    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd7a63a9-2208-46a6-99d6-1251e3a4ad17" containerName="revoke-keys"
Jul 17 11:46:43.218878 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.218598    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="573cbab5-8966-49fc-8b51-742a127177f7" containerName="revoke-keys"
Jul 17 11:46:43.218878 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.218604    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b17975d-97bf-4063-8f3d-6cbe41a0b476" containerName="revoke-keys"
Jul 17 11:46:43.218878 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.218609    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b17975d-97bf-4063-8f3d-6cbe41a0b476" containerName="revoke-keys"
Jul 17 11:46:43.221391 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.221374    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a50d5b-gxsxr"
Jul 17 11:46:43.223994 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.223969    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 11:46:43.224157 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.224027    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwv2h\""
Jul 17 11:46:43.235710 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.231597    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a50d5b-gxsxr"]
Jul 17 11:46:43.309464 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.309425    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2004a40-8448-48de-b065-3c5a61bf589c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a50d5b-gxsxr\" (UID: \"b2004a40-8448-48de-b065-3c5a61bf589c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a50d5b-gxsxr"
Jul 17 11:46:43.410528 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.410488    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2004a40-8448-48de-b065-3c5a61bf589c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a50d5b-gxsxr\" (UID: \"b2004a40-8448-48de-b065-3c5a61bf589c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a50d5b-gxsxr"
Jul 17 11:46:43.411149 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.411129    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2004a40-8448-48de-b065-3c5a61bf589c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a50d5b-gxsxr\" (UID: \"b2004a40-8448-48de-b065-3c5a61bf589c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a50d5b-gxsxr"
Jul 17 11:46:43.535198 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.535102    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a50d5b-gxsxr"
Jul 17 11:46:43.665906 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:43.665883    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a50d5b-gxsxr"]
Jul 17 11:46:43.667215 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:46:43.667189    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2004a40_8448_48de_b065_3c5a61bf589c.slice/crio-485f03e65193193dee48add187987ea8fb4634ebadc90e5bd56285c7c506b283 WatchSource:0}: Error finding container 485f03e65193193dee48add187987ea8fb4634ebadc90e5bd56285c7c506b283: Status 404 returned error can't find the container with id 485f03e65193193dee48add187987ea8fb4634ebadc90e5bd56285c7c506b283
Jul 17 11:46:44.001822 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:44.001773    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:44.101816 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:44.101785    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:44.612591 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:44.612559    2563 generic.go:356] "Generic (PLEG): container finished" podID="b2004a40-8448-48de-b065-3c5a61bf589c" containerID="788ff92b47065b1e562a1db32fbb7ca3eb928f0db0f4037cef85d197c7f68ee0" exitCode=0
Jul 17 11:46:44.612763 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:44.612602    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a50d5b-gxsxr" event={"ID":"b2004a40-8448-48de-b065-3c5a61bf589c","Type":"ContainerDied","Data":"788ff92b47065b1e562a1db32fbb7ca3eb928f0db0f4037cef85d197c7f68ee0"}
Jul 17 11:46:44.612763 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:44.612642    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a50d5b-gxsxr" event={"ID":"b2004a40-8448-48de-b065-3c5a61bf589c","Type":"ContainerStarted","Data":"485f03e65193193dee48add187987ea8fb4634ebadc90e5bd56285c7c506b283"}
Jul 17 11:46:45.747389 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:45.747367    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a50d5b-gxsxr"
Jul 17 11:46:45.830043 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:45.830013    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2004a40-8448-48de-b065-3c5a61bf589c-maas-api-service-ca\") pod \"b2004a40-8448-48de-b065-3c5a61bf589c\" (UID: \"b2004a40-8448-48de-b065-3c5a61bf589c\") "
Jul 17 11:46:45.830350 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:45.830329    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2004a40-8448-48de-b065-3c5a61bf589c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b2004a40-8448-48de-b065-3c5a61bf589c" (UID: "b2004a40-8448-48de-b065-3c5a61bf589c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:46:45.930896 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:45.930867    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2004a40-8448-48de-b065-3c5a61bf589c-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:46:46.624655 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:46.624622    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a50d5b-gxsxr"
Jul 17 11:46:46.624864 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:46.624657    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a50d5b-gxsxr" event={"ID":"b2004a40-8448-48de-b065-3c5a61bf589c","Type":"ContainerDied","Data":"485f03e65193193dee48add187987ea8fb4634ebadc90e5bd56285c7c506b283"}
Jul 17 11:46:46.624864 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:46.624700    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="485f03e65193193dee48add187987ea8fb4634ebadc90e5bd56285c7c506b283"
Jul 17 11:46:48.956012 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:48.955976    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-wt9pg"]
Jul 17 11:46:48.956416 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:48.956331    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2004a40-8448-48de-b065-3c5a61bf589c" containerName="revoke-keys"
Jul 17 11:46:48.956416 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:48.956345    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2004a40-8448-48de-b065-3c5a61bf589c" containerName="revoke-keys"
Jul 17 11:46:48.956416 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:48.956356    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="573cbab5-8966-49fc-8b51-742a127177f7" containerName="revoke-keys"
Jul 17 11:46:48.956416 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:48.956363    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="573cbab5-8966-49fc-8b51-742a127177f7" containerName="revoke-keys"
Jul 17 11:46:48.956556 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:48.956441    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="573cbab5-8966-49fc-8b51-742a127177f7" containerName="revoke-keys"
Jul 17 11:46:48.956556 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:48.956455    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2004a40-8448-48de-b065-3c5a61bf589c" containerName="revoke-keys"
Jul 17 11:46:48.958884 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:48.958868    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-wt9pg"
Jul 17 11:46:48.961465 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:48.961446    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwv2h\""
Jul 17 11:46:48.961548 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:48.961450    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 11:46:48.966457 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:48.966437    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-wt9pg"]
Jul 17 11:46:49.056830 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:49.056795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b56d7e-19d9-4fca-af19-8abceb7b93ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-40b693f1-wt9pg\" (UID: \"37b56d7e-19d9-4fca-af19-8abceb7b93ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-wt9pg"
Jul 17 11:46:49.157532 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:49.157501    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b56d7e-19d9-4fca-af19-8abceb7b93ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-40b693f1-wt9pg\" (UID: \"37b56d7e-19d9-4fca-af19-8abceb7b93ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-wt9pg"
Jul 17 11:46:49.158151 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:49.158133    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b56d7e-19d9-4fca-af19-8abceb7b93ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-40b693f1-wt9pg\" (UID: \"37b56d7e-19d9-4fca-af19-8abceb7b93ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-wt9pg"
Jul 17 11:46:49.270388 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:49.270290    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-wt9pg"
Jul 17 11:46:49.416614 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:49.416591    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-wt9pg"]
Jul 17 11:46:49.418629 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:46:49.418596    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b56d7e_19d9_4fca_af19_8abceb7b93ed.slice/crio-6b409cbbb3fd00f295c6705df5360890f7914a40fbb7453228fc1d6f1d674b72 WatchSource:0}: Error finding container 6b409cbbb3fd00f295c6705df5360890f7914a40fbb7453228fc1d6f1d674b72: Status 404 returned error can't find the container with id 6b409cbbb3fd00f295c6705df5360890f7914a40fbb7453228fc1d6f1d674b72
Jul 17 11:46:49.639410 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:49.639377    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-wt9pg" event={"ID":"37b56d7e-19d9-4fca-af19-8abceb7b93ed","Type":"ContainerStarted","Data":"6b409cbbb3fd00f295c6705df5360890f7914a40fbb7453228fc1d6f1d674b72"}
Jul 17 11:46:50.646945 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:50.646908    2563 generic.go:356] "Generic (PLEG): container finished" podID="37b56d7e-19d9-4fca-af19-8abceb7b93ed" containerID="909fa317aed2ca8ab75625785f651728b2577b44573266bbb5bcccc8905a5148" exitCode=6
Jul 17 11:46:50.647339 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:50.646988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-wt9pg" event={"ID":"37b56d7e-19d9-4fca-af19-8abceb7b93ed","Type":"ContainerDied","Data":"909fa317aed2ca8ab75625785f651728b2577b44573266bbb5bcccc8905a5148"}
Jul 17 11:46:50.647339 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:50.647219    2563 scope.go:117] "RemoveContainer" containerID="909fa317aed2ca8ab75625785f651728b2577b44573266bbb5bcccc8905a5148"
Jul 17 11:46:51.653385 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:51.653357    2563 generic.go:356] "Generic (PLEG): container finished" podID="37b56d7e-19d9-4fca-af19-8abceb7b93ed" containerID="57b93847eba80410dd94541e1923000efe85ddda1b34eec616b722890ec0c1f5" exitCode=6
Jul 17 11:46:51.653789 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:51.653434    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-wt9pg" event={"ID":"37b56d7e-19d9-4fca-af19-8abceb7b93ed","Type":"ContainerDied","Data":"57b93847eba80410dd94541e1923000efe85ddda1b34eec616b722890ec0c1f5"}
Jul 17 11:46:51.653789 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:51.653487    2563 scope.go:117] "RemoveContainer" containerID="909fa317aed2ca8ab75625785f651728b2577b44573266bbb5bcccc8905a5148"
Jul 17 11:46:51.653789 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:51.653656    2563 scope.go:117] "RemoveContainer" containerID="57b93847eba80410dd94541e1923000efe85ddda1b34eec616b722890ec0c1f5"
Jul 17 11:46:51.653914 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:46:51.653876    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-40b693f1-wt9pg_odh-ai-gateway-infra(37b56d7e-19d9-4fca-af19-8abceb7b93ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-wt9pg" podUID="37b56d7e-19d9-4fca-af19-8abceb7b93ed"
Jul 17 11:46:52.660956 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:52.660933    2563 scope.go:117] "RemoveContainer" containerID="57b93847eba80410dd94541e1923000efe85ddda1b34eec616b722890ec0c1f5"
Jul 17 11:46:52.661407 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:46:52.661187    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-40b693f1-wt9pg_odh-ai-gateway-infra(37b56d7e-19d9-4fca-af19-8abceb7b93ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-wt9pg" podUID="37b56d7e-19d9-4fca-af19-8abceb7b93ed"
Jul 17 11:46:53.361624 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:53.361589    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:53.458711 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:53.458680    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:53.675282 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:53.675244    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:46:58.331737 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:46:58.331697    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:04.762838 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:04.762800    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:04.876694 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:04.876658    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:07.660233 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:07.660204    2563 scope.go:117] "RemoveContainer" containerID="57b93847eba80410dd94541e1923000efe85ddda1b34eec616b722890ec0c1f5"
Jul 17 11:47:08.754534 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:08.754501    2563 generic.go:356] "Generic (PLEG): container finished" podID="37b56d7e-19d9-4fca-af19-8abceb7b93ed" containerID="21d13702737716f1e10890c726f8e2ec5a0d933ec7634a2f51aec835d83a9a9c" exitCode=6
Jul 17 11:47:08.754975 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:08.754550    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-wt9pg" event={"ID":"37b56d7e-19d9-4fca-af19-8abceb7b93ed","Type":"ContainerDied","Data":"21d13702737716f1e10890c726f8e2ec5a0d933ec7634a2f51aec835d83a9a9c"}
Jul 17 11:47:08.754975 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:08.754596    2563 scope.go:117] "RemoveContainer" containerID="57b93847eba80410dd94541e1923000efe85ddda1b34eec616b722890ec0c1f5"
Jul 17 11:47:08.754975 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:08.754847    2563 scope.go:117] "RemoveContainer" containerID="21d13702737716f1e10890c726f8e2ec5a0d933ec7634a2f51aec835d83a9a9c"
Jul 17 11:47:08.755149 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:47:08.755117    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-40b693f1-wt9pg_odh-ai-gateway-infra(37b56d7e-19d9-4fca-af19-8abceb7b93ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-wt9pg" podUID="37b56d7e-19d9-4fca-af19-8abceb7b93ed"
Jul 17 11:47:09.112040 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:09.111949    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:09.387967 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:09.387923    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:09.782996 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:09.782915    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-wt9pg"]
Jul 17 11:47:09.807086 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:09.807040    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:09.914120 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:09.914097    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-wt9pg"
Jul 17 11:47:10.046558 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.046478    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b56d7e-19d9-4fca-af19-8abceb7b93ed-maas-api-service-ca\") pod \"37b56d7e-19d9-4fca-af19-8abceb7b93ed\" (UID: \"37b56d7e-19d9-4fca-af19-8abceb7b93ed\") "
Jul 17 11:47:10.046817 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.046796    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b56d7e-19d9-4fca-af19-8abceb7b93ed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "37b56d7e-19d9-4fca-af19-8abceb7b93ed" (UID: "37b56d7e-19d9-4fca-af19-8abceb7b93ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:47:10.140443 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.140402    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"]
Jul 17 11:47:10.140798 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.140782    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37b56d7e-19d9-4fca-af19-8abceb7b93ed" containerName="revoke-keys"
Jul 17 11:47:10.140881 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.140802    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="37b56d7e-19d9-4fca-af19-8abceb7b93ed" containerName="revoke-keys"
Jul 17 11:47:10.140881 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.140845    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37b56d7e-19d9-4fca-af19-8abceb7b93ed" containerName="revoke-keys"
Jul 17 11:47:10.140881 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.140854    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="37b56d7e-19d9-4fca-af19-8abceb7b93ed" containerName="revoke-keys"
Jul 17 11:47:10.141029 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.140934    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="37b56d7e-19d9-4fca-af19-8abceb7b93ed" containerName="revoke-keys"
Jul 17 11:47:10.141029 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.140947    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="37b56d7e-19d9-4fca-af19-8abceb7b93ed" containerName="revoke-keys"
Jul 17 11:47:10.143916 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.143895    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.146119 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.146087    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:10.146912 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.146895    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-ad4893-openshift-default-dockercfg-njf5x\""
Jul 17 11:47:10.147290 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.147273    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b56d7e-19d9-4fca-af19-8abceb7b93ed-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:47:10.159029 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.159004    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"]
Jul 17 11:47:10.247611 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.247576    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:10.247764 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.247699    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istio-envoy\") pod \"e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") " pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.247764 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.247727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istio-podinfo\") pod \"e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") " pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.247764 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.247762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-workload-socket\") pod \"e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") " pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.247923 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.247803    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6cj2\" (UniqueName: \"kubernetes.io/projected/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-kube-api-access-m6cj2\") pod \"e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") " pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.247923 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.247827    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-workload-certs\") pod \"e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") " pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.247923 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.247844    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istio-data\") pod \"e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") " pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.247923 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.247860    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istio-token\") pod \"e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") " pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.247923 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.247879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istiod-ca-cert\") pod \"e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") " pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.248179 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.247975    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-credential-socket\") pod \"e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") " pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.349508 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.349426    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6cj2\" (UniqueName: \"kubernetes.io/projected/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-kube-api-access-m6cj2\") pod \"e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") " pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.349508 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.349462    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-workload-certs\") pod \"e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") " pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.349508 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.349477    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istio-data\") pod \"e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") " pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.349508 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.349493    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istio-token\") pod \"e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") " pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.349841 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.349515    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istiod-ca-cert\") pod \"e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") " pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.349841 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.349590    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-credential-socket\") pod \"e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") " pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.349841 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.349633    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istio-envoy\") pod \"e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") " pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.349841 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.349654    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istio-podinfo\") pod \"e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") " pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.349841 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.349747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-workload-socket\") pod \"e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") " pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.350172 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.349894    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-workload-certs\") pod \"e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") " pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.350172 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.349937    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istio-data\") pod \"e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") " pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.350172 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.350128    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-workload-socket\") pod \"e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") " pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.350346 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.350254    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-credential-socket\") pod \"e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") " pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.350468 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.350444    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istiod-ca-cert\") pod \"e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") " pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.351902 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.351875    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istio-envoy\") pod \"e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") " pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.352182 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.352162    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istio-podinfo\") pod \"e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") " pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.359540 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.359521    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istio-token\") pod \"e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") " pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.359948 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.359929    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6cj2\" (UniqueName: \"kubernetes.io/projected/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-kube-api-access-m6cj2\") pod \"e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") " pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.454806 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.454775    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:10.590531 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.590504    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"]
Jul 17 11:47:10.591804 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:47:10.591781    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1b08e5_60d6_4fdc_83eb_d5365a9d0661.slice/crio-68d019a891b92714c3e58ac4c036375764250992f7f8a25a3667c1995eab1dba WatchSource:0}: Error finding container 68d019a891b92714c3e58ac4c036375764250992f7f8a25a3667c1995eab1dba: Status 404 returned error can't find the container with id 68d019a891b92714c3e58ac4c036375764250992f7f8a25a3667c1995eab1dba
Jul 17 11:47:10.593858 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.593825    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 17 11:47:10.593976 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.593903    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 17 11:47:10.593976 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.593956    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 17 11:47:10.774782 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.774748    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-wt9pg"
Jul 17 11:47:10.774782 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.774761    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-wt9pg" event={"ID":"37b56d7e-19d9-4fca-af19-8abceb7b93ed","Type":"ContainerDied","Data":"6b409cbbb3fd00f295c6705df5360890f7914a40fbb7453228fc1d6f1d674b72"}
Jul 17 11:47:10.775011 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.774860    2563 scope.go:117] "RemoveContainer" containerID="21d13702737716f1e10890c726f8e2ec5a0d933ec7634a2f51aec835d83a9a9c"
Jul 17 11:47:10.776857 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.776828    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh" event={"ID":"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661","Type":"ContainerStarted","Data":"eaf5710e8454e1fbd311b9cee2ae2f1ddb15a7b751713e855ef408aaad35dac8"}
Jul 17 11:47:10.776983 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.776878    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh" event={"ID":"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661","Type":"ContainerStarted","Data":"68d019a891b92714c3e58ac4c036375764250992f7f8a25a3667c1995eab1dba"}
Jul 17 11:47:10.802746 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.802694    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh" podStartSLOduration=0.802679985 podStartE2EDuration="802.679985ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:47:10.802118258 +0000 UTC m=+2985.731657899" watchObservedRunningTime="2026-07-17 11:47:10.802679985 +0000 UTC m=+2985.732219625"
Jul 17 11:47:10.817753 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.817736    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-wt9pg"]
Jul 17 11:47:10.820442 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:10.820420    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-40b693f1-wt9pg"]
Jul 17 11:47:11.454993 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:11.454957    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:11.460125 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:11.460105    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:11.665969 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:11.665921    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37b56d7e-19d9-4fca-af19-8abceb7b93ed" path="/var/lib/kubelet/pods/37b56d7e-19d9-4fca-af19-8abceb7b93ed/volumes"
Jul 17 11:47:11.783501 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:11.783426    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:11.784288 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:11.784266    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:12.823356 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:12.823313    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:12.937552 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:12.937517    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:13.039289 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:13.039250    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:13.531126 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:13.531062    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:13.630650 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:13.630611    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:17.384943 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:17.384903    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:17.770336 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:17.770248    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:19.212114 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:19.212061    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:20.754208 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:20.754171    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:20.860660 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:20.860627    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:20.971629 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:20.971594    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:21.227417 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:21.227381    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:21.455359 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:21.455302    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:21.552109 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:21.551994    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:25.301920 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:25.301880    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:25.801632 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:25.801612    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nj5mj_55d1de37-8635-4de8-ab59-4ebfd0346a40/ovn-acl-logging/0.log"
Jul 17 11:47:25.815215 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:25.815193    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nj5mj_55d1de37-8635-4de8-ab59-4ebfd0346a40/ovn-acl-logging/0.log"
Jul 17 11:47:25.857533 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:25.857502    2563 generic.go:356] "Generic (PLEG): container finished" podID="9c1b08e5-60d6-4fdc-83eb-d5365a9d0661" containerID="eaf5710e8454e1fbd311b9cee2ae2f1ddb15a7b751713e855ef408aaad35dac8" exitCode=0
Jul 17 11:47:25.857661 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:25.857571    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh" event={"ID":"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661","Type":"ContainerDied","Data":"eaf5710e8454e1fbd311b9cee2ae2f1ddb15a7b751713e855ef408aaad35dac8"}
Jul 17 11:47:25.857982 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:25.857957    2563 scope.go:117] "RemoveContainer" containerID="eaf5710e8454e1fbd311b9cee2ae2f1ddb15a7b751713e855ef408aaad35dac8"
Jul 17 11:47:25.858894 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:25.858871    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 17 11:47:25.858944 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:25.858930    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 17 11:47:25.858993 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:25.858959    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 17 11:47:26.783686 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:26.783653    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:26.863604 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:26.863575    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh" event={"ID":"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661","Type":"ContainerStarted","Data":"fe1da9c6f7a0707767bea652e22ad83b5f2721fea3a76422068da459b8796b17"}
Jul 17 11:47:27.455404 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:27.455349    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:27.456440 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:27.456416    2563 patch_prober.go:28] interesting pod/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Jul 17 11:47:27.456562 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:27.456460    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh" podUID="9c1b08e5-60d6-4fdc-83eb-d5365a9d0661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Jul 17 11:47:28.455552 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:28.455516    2563 patch_prober.go:28] interesting pod/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Jul 17 11:47:28.455934 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:28.455571    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh" podUID="9c1b08e5-60d6-4fdc-83eb-d5365a9d0661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Jul 17 11:47:29.455367 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:29.455332    2563 patch_prober.go:28] interesting pod/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Jul 17 11:47:29.455544 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:29.455385    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh" podUID="9c1b08e5-60d6-4fdc-83eb-d5365a9d0661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Jul 17 11:47:30.455648 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:30.455597    2563 patch_prober.go:28] interesting pod/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Jul 17 11:47:30.456183 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:30.455669    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh" podUID="9c1b08e5-60d6-4fdc-83eb-d5365a9d0661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Jul 17 11:47:31.455917 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:31.455813    2563 patch_prober.go:28] interesting pod/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Jul 17 11:47:31.456409 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:31.455919    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh" podUID="9c1b08e5-60d6-4fdc-83eb-d5365a9d0661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Jul 17 11:47:32.455591 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:32.455552    2563 patch_prober.go:28] interesting pod/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Jul 17 11:47:32.455908 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:32.455876    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh" podUID="9c1b08e5-60d6-4fdc-83eb-d5365a9d0661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Jul 17 11:47:33.455403 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:33.455362    2563 patch_prober.go:28] interesting pod/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Jul 17 11:47:33.456033 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:33.456000    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh" podUID="9c1b08e5-60d6-4fdc-83eb-d5365a9d0661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Jul 17 11:47:34.467416 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:34.467380    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:34.467900 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:34.467733    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:34.468566 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:34.468546    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:47:34.567164 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:34.567116    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:34.676118 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:34.676058    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:42.290995 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:42.290952    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:42.391570 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:42.391530    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:45.193474 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.193442    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"]
Jul 17 11:47:45.193860 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.193782    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37b56d7e-19d9-4fca-af19-8abceb7b93ed" containerName="revoke-keys"
Jul 17 11:47:45.193860 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.193793    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="37b56d7e-19d9-4fca-af19-8abceb7b93ed" containerName="revoke-keys"
Jul 17 11:47:45.193947 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.193880    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="37b56d7e-19d9-4fca-af19-8abceb7b93ed" containerName="revoke-keys"
Jul 17 11:47:45.195939 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.195924    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"
Jul 17 11:47:45.199916 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.199893    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ad4893\"/\"default-dockercfg-9dnx7\""
Jul 17 11:47:45.200127 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.199941    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ad4893\"/\"kube-root-ca.crt\""
Jul 17 11:47:45.200127 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.199953    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ad4893\"/\"openshift-service-ca.crt\""
Jul 17 11:47:45.200127 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.199953    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ad4893\"/\"auth-test-model-ad4893-kserve-self-signed-certs\""
Jul 17 11:47:45.208646 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.208624    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"]
Jul 17 11:47:45.231892 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.231867    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:47:45.273691 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.273669    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c28fe762-9355-42b7-a514-f809873833bb-tls-certs\") pod \"auth-test-model-ad4893-kserve-7666559899-z8t85\" (UID: \"c28fe762-9355-42b7-a514-f809873833bb\") " pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"
Jul 17 11:47:45.273804 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.273712    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c28fe762-9355-42b7-a514-f809873833bb-tmp-dir\") pod \"auth-test-model-ad4893-kserve-7666559899-z8t85\" (UID: \"c28fe762-9355-42b7-a514-f809873833bb\") " pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"
Jul 17 11:47:45.273804 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.273738    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c28fe762-9355-42b7-a514-f809873833bb-dshm\") pod \"auth-test-model-ad4893-kserve-7666559899-z8t85\" (UID: \"c28fe762-9355-42b7-a514-f809873833bb\") " pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"
Jul 17 11:47:45.273804 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.273764    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c28fe762-9355-42b7-a514-f809873833bb-home\") pod \"auth-test-model-ad4893-kserve-7666559899-z8t85\" (UID: \"c28fe762-9355-42b7-a514-f809873833bb\") " pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"
Jul 17 11:47:45.273804 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.273780    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2k7k\" (UniqueName: \"kubernetes.io/projected/c28fe762-9355-42b7-a514-f809873833bb-kube-api-access-k2k7k\") pod \"auth-test-model-ad4893-kserve-7666559899-z8t85\" (UID: \"c28fe762-9355-42b7-a514-f809873833bb\") " pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"
Jul 17 11:47:45.273968 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.273840    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c28fe762-9355-42b7-a514-f809873833bb-model-cache\") pod \"auth-test-model-ad4893-kserve-7666559899-z8t85\" (UID: \"c28fe762-9355-42b7-a514-f809873833bb\") " pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"
Jul 17 11:47:45.375005 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.374973    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c28fe762-9355-42b7-a514-f809873833bb-model-cache\") pod \"auth-test-model-ad4893-kserve-7666559899-z8t85\" (UID: \"c28fe762-9355-42b7-a514-f809873833bb\") " pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"
Jul 17 11:47:45.375206 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.375017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c28fe762-9355-42b7-a514-f809873833bb-tls-certs\") pod \"auth-test-model-ad4893-kserve-7666559899-z8t85\" (UID: \"c28fe762-9355-42b7-a514-f809873833bb\") " pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"
Jul 17 11:47:45.375206 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.375063    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c28fe762-9355-42b7-a514-f809873833bb-tmp-dir\") pod \"auth-test-model-ad4893-kserve-7666559899-z8t85\" (UID: \"c28fe762-9355-42b7-a514-f809873833bb\") " pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"
Jul 17 11:47:45.375206 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.375107    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c28fe762-9355-42b7-a514-f809873833bb-dshm\") pod \"auth-test-model-ad4893-kserve-7666559899-z8t85\" (UID: \"c28fe762-9355-42b7-a514-f809873833bb\") " pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"
Jul 17 11:47:45.375206 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.375134    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c28fe762-9355-42b7-a514-f809873833bb-home\") pod \"auth-test-model-ad4893-kserve-7666559899-z8t85\" (UID: \"c28fe762-9355-42b7-a514-f809873833bb\") " pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"
Jul 17 11:47:45.375206 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.375157    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2k7k\" (UniqueName: \"kubernetes.io/projected/c28fe762-9355-42b7-a514-f809873833bb-kube-api-access-k2k7k\") pod \"auth-test-model-ad4893-kserve-7666559899-z8t85\" (UID: \"c28fe762-9355-42b7-a514-f809873833bb\") " pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"
Jul 17 11:47:45.375512 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.375483    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c28fe762-9355-42b7-a514-f809873833bb-model-cache\") pod \"auth-test-model-ad4893-kserve-7666559899-z8t85\" (UID: \"c28fe762-9355-42b7-a514-f809873833bb\") " pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"
Jul 17 11:47:45.375638 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.375486    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c28fe762-9355-42b7-a514-f809873833bb-tmp-dir\") pod \"auth-test-model-ad4893-kserve-7666559899-z8t85\" (UID: \"c28fe762-9355-42b7-a514-f809873833bb\") " pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"
Jul 17 11:47:45.375638 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.375560    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c28fe762-9355-42b7-a514-f809873833bb-home\") pod \"auth-test-model-ad4893-kserve-7666559899-z8t85\" (UID: \"c28fe762-9355-42b7-a514-f809873833bb\") " pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"
Jul 17 11:47:45.377281 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.377255    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c28fe762-9355-42b7-a514-f809873833bb-dshm\") pod \"auth-test-model-ad4893-kserve-7666559899-z8t85\" (UID: \"c28fe762-9355-42b7-a514-f809873833bb\") " pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"
Jul 17 11:47:45.377578 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.377560    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c28fe762-9355-42b7-a514-f809873833bb-tls-certs\") pod \"auth-test-model-ad4893-kserve-7666559899-z8t85\" (UID: \"c28fe762-9355-42b7-a514-f809873833bb\") " pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"
Jul 17 11:47:45.386561 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.386541    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2k7k\" (UniqueName: \"kubernetes.io/projected/c28fe762-9355-42b7-a514-f809873833bb-kube-api-access-k2k7k\") pod \"auth-test-model-ad4893-kserve-7666559899-z8t85\" (UID: \"c28fe762-9355-42b7-a514-f809873833bb\") " pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"
Jul 17 11:47:45.505476 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.505384    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"
Jul 17 11:47:45.636789 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.636763    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"]
Jul 17 11:47:45.638723 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:47:45.638693    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc28fe762_9355_42b7_a514_f809873833bb.slice/crio-d6820f73cc6a22a693fe3448a1a3252c0dca176d96186664f58981decc64368f WatchSource:0}: Error finding container d6820f73cc6a22a693fe3448a1a3252c0dca176d96186664f58981decc64368f: Status 404 returned error can't find the container with id d6820f73cc6a22a693fe3448a1a3252c0dca176d96186664f58981decc64368f
Jul 17 11:47:45.964925 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.964888    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85" event={"ID":"c28fe762-9355-42b7-a514-f809873833bb","Type":"ContainerStarted","Data":"d6a39b3dda4373e068b9803da7ede61b6a9fcab59273fb094816f768d70d306f"}
Jul 17 11:47:45.965118 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:45.964935    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85" event={"ID":"c28fe762-9355-42b7-a514-f809873833bb","Type":"ContainerStarted","Data":"d6820f73cc6a22a693fe3448a1a3252c0dca176d96186664f58981decc64368f"}
Jul 17 11:47:46.000289 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:46.000246    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85" podStartSLOduration=1.000230305 podStartE2EDuration="1.000230305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:47:45.999653449 +0000 UTC m=+3020.929193090" watchObservedRunningTime="2026-07-17 11:47:46.000230305 +0000 UTC m=+3020.929769987"
Jul 17 11:47:55.505692 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:55.505655    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"
Jul 17 11:47:55.506148 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:55.505710    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"
Jul 17 11:47:55.518255 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:55.518229    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"
Jul 17 11:47:56.028209 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:47:56.028177    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"
Jul 17 11:48:06.300014 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.299982    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"]
Jul 17 11:48:06.303261 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.303241    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"
Jul 17 11:48:06.307005 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.306980    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9d836f\"/\"kube-root-ca.crt\""
Jul 17 11:48:06.307180 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.307010    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9d836f\"/\"default-dockercfg-mrs27\""
Jul 17 11:48:06.307180 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.307037    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9d836f\"/\"auth-test-model-9d836f-kserve-self-signed-certs\""
Jul 17 11:48:06.307180 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.307091    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9d836f\"/\"openshift-service-ca.crt\""
Jul 17 11:48:06.313289 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.313268    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"]
Jul 17 11:48:06.339106 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.339061    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:48:06.466690 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.466646    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b842528-e00f-464c-8dab-81400143fec7-home\") pod \"auth-test-model-9d836f-kserve-7cbf4795f9-zggjz\" (UID: \"1b842528-e00f-464c-8dab-81400143fec7\") " pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"
Jul 17 11:48:06.466690 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.466689    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b842528-e00f-464c-8dab-81400143fec7-tls-certs\") pod \"auth-test-model-9d836f-kserve-7cbf4795f9-zggjz\" (UID: \"1b842528-e00f-464c-8dab-81400143fec7\") " pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"
Jul 17 11:48:06.466911 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.466782    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b842528-e00f-464c-8dab-81400143fec7-tmp-dir\") pod \"auth-test-model-9d836f-kserve-7cbf4795f9-zggjz\" (UID: \"1b842528-e00f-464c-8dab-81400143fec7\") " pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"
Jul 17 11:48:06.466911 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.466826    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b842528-e00f-464c-8dab-81400143fec7-model-cache\") pod \"auth-test-model-9d836f-kserve-7cbf4795f9-zggjz\" (UID: \"1b842528-e00f-464c-8dab-81400143fec7\") " pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"
Jul 17 11:48:06.466911 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.466849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b842528-e00f-464c-8dab-81400143fec7-dshm\") pod \"auth-test-model-9d836f-kserve-7cbf4795f9-zggjz\" (UID: \"1b842528-e00f-464c-8dab-81400143fec7\") " pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"
Jul 17 11:48:06.466911 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.466907    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2gvc\" (UniqueName: \"kubernetes.io/projected/1b842528-e00f-464c-8dab-81400143fec7-kube-api-access-n2gvc\") pod \"auth-test-model-9d836f-kserve-7cbf4795f9-zggjz\" (UID: \"1b842528-e00f-464c-8dab-81400143fec7\") " pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"
Jul 17 11:48:06.568243 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.568152    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b842528-e00f-464c-8dab-81400143fec7-model-cache\") pod \"auth-test-model-9d836f-kserve-7cbf4795f9-zggjz\" (UID: \"1b842528-e00f-464c-8dab-81400143fec7\") " pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"
Jul 17 11:48:06.568243 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.568193    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b842528-e00f-464c-8dab-81400143fec7-dshm\") pod \"auth-test-model-9d836f-kserve-7cbf4795f9-zggjz\" (UID: \"1b842528-e00f-464c-8dab-81400143fec7\") " pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"
Jul 17 11:48:06.568479 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.568319    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2gvc\" (UniqueName: \"kubernetes.io/projected/1b842528-e00f-464c-8dab-81400143fec7-kube-api-access-n2gvc\") pod \"auth-test-model-9d836f-kserve-7cbf4795f9-zggjz\" (UID: \"1b842528-e00f-464c-8dab-81400143fec7\") " pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"
Jul 17 11:48:06.568479 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.568377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b842528-e00f-464c-8dab-81400143fec7-home\") pod \"auth-test-model-9d836f-kserve-7cbf4795f9-zggjz\" (UID: \"1b842528-e00f-464c-8dab-81400143fec7\") " pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"
Jul 17 11:48:06.568479 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.568397    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b842528-e00f-464c-8dab-81400143fec7-tls-certs\") pod \"auth-test-model-9d836f-kserve-7cbf4795f9-zggjz\" (UID: \"1b842528-e00f-464c-8dab-81400143fec7\") " pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"
Jul 17 11:48:06.568479 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.568461    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b842528-e00f-464c-8dab-81400143fec7-tmp-dir\") pod \"auth-test-model-9d836f-kserve-7cbf4795f9-zggjz\" (UID: \"1b842528-e00f-464c-8dab-81400143fec7\") " pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"
Jul 17 11:48:06.568694 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.568669    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b842528-e00f-464c-8dab-81400143fec7-model-cache\") pod \"auth-test-model-9d836f-kserve-7cbf4795f9-zggjz\" (UID: \"1b842528-e00f-464c-8dab-81400143fec7\") " pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"
Jul 17 11:48:06.568749 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.568695    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b842528-e00f-464c-8dab-81400143fec7-home\") pod \"auth-test-model-9d836f-kserve-7cbf4795f9-zggjz\" (UID: \"1b842528-e00f-464c-8dab-81400143fec7\") " pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"
Jul 17 11:48:06.568831 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.568808    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b842528-e00f-464c-8dab-81400143fec7-tmp-dir\") pod \"auth-test-model-9d836f-kserve-7cbf4795f9-zggjz\" (UID: \"1b842528-e00f-464c-8dab-81400143fec7\") " pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"
Jul 17 11:48:06.570475 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.570447    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b842528-e00f-464c-8dab-81400143fec7-dshm\") pod \"auth-test-model-9d836f-kserve-7cbf4795f9-zggjz\" (UID: \"1b842528-e00f-464c-8dab-81400143fec7\") " pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"
Jul 17 11:48:06.570731 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.570715    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b842528-e00f-464c-8dab-81400143fec7-tls-certs\") pod \"auth-test-model-9d836f-kserve-7cbf4795f9-zggjz\" (UID: \"1b842528-e00f-464c-8dab-81400143fec7\") " pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"
Jul 17 11:48:06.578608 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.578584    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2gvc\" (UniqueName: \"kubernetes.io/projected/1b842528-e00f-464c-8dab-81400143fec7-kube-api-access-n2gvc\") pod \"auth-test-model-9d836f-kserve-7cbf4795f9-zggjz\" (UID: \"1b842528-e00f-464c-8dab-81400143fec7\") " pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"
Jul 17 11:48:06.614027 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.614005    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"
Jul 17 11:48:06.745488 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:06.745462    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"]
Jul 17 11:48:06.747505 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:48:06.747470    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b842528_e00f_464c_8dab_81400143fec7.slice/crio-485dc1eaba45f53c4a10621637e121497bf87680d1ad0e9e8b3a3e25f93f8947 WatchSource:0}: Error finding container 485dc1eaba45f53c4a10621637e121497bf87680d1ad0e9e8b3a3e25f93f8947: Status 404 returned error can't find the container with id 485dc1eaba45f53c4a10621637e121497bf87680d1ad0e9e8b3a3e25f93f8947
Jul 17 11:48:07.075995 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:07.075958    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz" event={"ID":"1b842528-e00f-464c-8dab-81400143fec7","Type":"ContainerStarted","Data":"e15d47c493b9869b60d81407cb6d78cd68dd91f86ac209e93faa84bfa5662bb2"}
Jul 17 11:48:07.075995 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:07.076002    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz" event={"ID":"1b842528-e00f-464c-8dab-81400143fec7","Type":"ContainerStarted","Data":"485dc1eaba45f53c4a10621637e121497bf87680d1ad0e9e8b3a3e25f93f8947"}
Jul 17 11:48:07.095792 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:07.095748    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz" podStartSLOduration=1.095733281 podStartE2EDuration="1.095733281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:48:07.094323463 +0000 UTC m=+3042.023863106" watchObservedRunningTime="2026-07-17 11:48:07.095733281 +0000 UTC m=+3042.025272921"
Jul 17 11:48:16.614358 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:16.614319    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"
Jul 17 11:48:16.614358 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:16.614369    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"
Jul 17 11:48:16.626881 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:16.626858    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"
Jul 17 11:48:17.144560 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:48:17.144529    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"
Jul 17 11:50:07.811537 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:07.811459    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"]
Jul 17 11:50:07.870613 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:07.870566    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:50:08.567889 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:08.567859    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"]
Jul 17 11:50:08.589241 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:08.589204    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:50:18.986774 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:18.986734    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:50:22.830951 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:22.830908    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85" podUID="c28fe762-9355-42b7-a514-f809873833bb" containerName="main" containerID="cri-o://d6a39b3dda4373e068b9803da7ede61b6a9fcab59273fb094816f768d70d306f" gracePeriod=45
Jul 17 11:50:23.092401 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.092331    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"
Jul 17 11:50:23.198709 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.198673    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c28fe762-9355-42b7-a514-f809873833bb-tls-certs\") pod \"c28fe762-9355-42b7-a514-f809873833bb\" (UID: \"c28fe762-9355-42b7-a514-f809873833bb\") "
Jul 17 11:50:23.198900 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.198726    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c28fe762-9355-42b7-a514-f809873833bb-dshm\") pod \"c28fe762-9355-42b7-a514-f809873833bb\" (UID: \"c28fe762-9355-42b7-a514-f809873833bb\") "
Jul 17 11:50:23.198900 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.198835    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2k7k\" (UniqueName: \"kubernetes.io/projected/c28fe762-9355-42b7-a514-f809873833bb-kube-api-access-k2k7k\") pod \"c28fe762-9355-42b7-a514-f809873833bb\" (UID: \"c28fe762-9355-42b7-a514-f809873833bb\") "
Jul 17 11:50:23.198900 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.198894    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c28fe762-9355-42b7-a514-f809873833bb-tmp-dir\") pod \"c28fe762-9355-42b7-a514-f809873833bb\" (UID: \"c28fe762-9355-42b7-a514-f809873833bb\") "
Jul 17 11:50:23.199095 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.198954    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c28fe762-9355-42b7-a514-f809873833bb-home\") pod \"c28fe762-9355-42b7-a514-f809873833bb\" (UID: \"c28fe762-9355-42b7-a514-f809873833bb\") "
Jul 17 11:50:23.199095 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.198977    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c28fe762-9355-42b7-a514-f809873833bb-model-cache\") pod \"c28fe762-9355-42b7-a514-f809873833bb\" (UID: \"c28fe762-9355-42b7-a514-f809873833bb\") "
Jul 17 11:50:23.199225 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.199175    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c28fe762-9355-42b7-a514-f809873833bb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c28fe762-9355-42b7-a514-f809873833bb" (UID: "c28fe762-9355-42b7-a514-f809873833bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:50:23.199305 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.199261    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c28fe762-9355-42b7-a514-f809873833bb-home" (OuterVolumeSpecName: "home") pod "c28fe762-9355-42b7-a514-f809873833bb" (UID: "c28fe762-9355-42b7-a514-f809873833bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:50:23.199382 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.199315    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c28fe762-9355-42b7-a514-f809873833bb-model-cache" (OuterVolumeSpecName: "model-cache") pod "c28fe762-9355-42b7-a514-f809873833bb" (UID: "c28fe762-9355-42b7-a514-f809873833bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:50:23.199437 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.199406    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c28fe762-9355-42b7-a514-f809873833bb-tmp-dir\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:50:23.199437 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.199426    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c28fe762-9355-42b7-a514-f809873833bb-home\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:50:23.199581 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.199440    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c28fe762-9355-42b7-a514-f809873833bb-model-cache\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:50:23.201484 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.201449    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c28fe762-9355-42b7-a514-f809873833bb-dshm" (OuterVolumeSpecName: "dshm") pod "c28fe762-9355-42b7-a514-f809873833bb" (UID: "c28fe762-9355-42b7-a514-f809873833bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:50:23.201598 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.201532    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c28fe762-9355-42b7-a514-f809873833bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c28fe762-9355-42b7-a514-f809873833bb" (UID: "c28fe762-9355-42b7-a514-f809873833bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 11:50:23.201598 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.201557    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c28fe762-9355-42b7-a514-f809873833bb-kube-api-access-k2k7k" (OuterVolumeSpecName: "kube-api-access-k2k7k") pod "c28fe762-9355-42b7-a514-f809873833bb" (UID: "c28fe762-9355-42b7-a514-f809873833bb"). InnerVolumeSpecName "kube-api-access-k2k7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 11:50:23.300498 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.300459    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c28fe762-9355-42b7-a514-f809873833bb-tls-certs\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:50:23.300498 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.300496    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c28fe762-9355-42b7-a514-f809873833bb-dshm\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:50:23.300498 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.300506    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2k7k\" (UniqueName: \"kubernetes.io/projected/c28fe762-9355-42b7-a514-f809873833bb-kube-api-access-k2k7k\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:50:23.587530 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.587493    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz" podUID="1b842528-e00f-464c-8dab-81400143fec7" containerName="main" containerID="cri-o://e15d47c493b9869b60d81407cb6d78cd68dd91f86ac209e93faa84bfa5662bb2" gracePeriod=45
Jul 17 11:50:23.801464 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.801431    2563 generic.go:356] "Generic (PLEG): container finished" podID="1b842528-e00f-464c-8dab-81400143fec7" containerID="e15d47c493b9869b60d81407cb6d78cd68dd91f86ac209e93faa84bfa5662bb2" exitCode=0
Jul 17 11:50:23.801647 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.801500    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz" event={"ID":"1b842528-e00f-464c-8dab-81400143fec7","Type":"ContainerDied","Data":"e15d47c493b9869b60d81407cb6d78cd68dd91f86ac209e93faa84bfa5662bb2"}
Jul 17 11:50:23.802856 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.802830    2563 generic.go:356] "Generic (PLEG): container finished" podID="c28fe762-9355-42b7-a514-f809873833bb" containerID="d6a39b3dda4373e068b9803da7ede61b6a9fcab59273fb094816f768d70d306f" exitCode=0
Jul 17 11:50:23.802975 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.802871    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85" event={"ID":"c28fe762-9355-42b7-a514-f809873833bb","Type":"ContainerDied","Data":"d6a39b3dda4373e068b9803da7ede61b6a9fcab59273fb094816f768d70d306f"}
Jul 17 11:50:23.802975 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.802919    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85" event={"ID":"c28fe762-9355-42b7-a514-f809873833bb","Type":"ContainerDied","Data":"d6820f73cc6a22a693fe3448a1a3252c0dca176d96186664f58981decc64368f"}
Jul 17 11:50:23.802975 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.802939    2563 scope.go:117] "RemoveContainer" containerID="d6a39b3dda4373e068b9803da7ede61b6a9fcab59273fb094816f768d70d306f"
Jul 17 11:50:23.802975 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.802875    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"
Jul 17 11:50:23.812976 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.812954    2563 scope.go:117] "RemoveContainer" containerID="d6a39b3dda4373e068b9803da7ede61b6a9fcab59273fb094816f768d70d306f"
Jul 17 11:50:23.813242 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:50:23.813221    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6a39b3dda4373e068b9803da7ede61b6a9fcab59273fb094816f768d70d306f\": container with ID starting with d6a39b3dda4373e068b9803da7ede61b6a9fcab59273fb094816f768d70d306f not found: ID does not exist" containerID="d6a39b3dda4373e068b9803da7ede61b6a9fcab59273fb094816f768d70d306f"
Jul 17 11:50:23.813298 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.813254    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6a39b3dda4373e068b9803da7ede61b6a9fcab59273fb094816f768d70d306f"} err="failed to get container status \"d6a39b3dda4373e068b9803da7ede61b6a9fcab59273fb094816f768d70d306f\": rpc error: code = NotFound desc = could not find container \"d6a39b3dda4373e068b9803da7ede61b6a9fcab59273fb094816f768d70d306f\": container with ID starting with d6a39b3dda4373e068b9803da7ede61b6a9fcab59273fb094816f768d70d306f not found: ID does not exist"
Jul 17 11:50:23.822733 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.822684    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"
Jul 17 11:50:23.823527 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.823508    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"]
Jul 17 11:50:23.825748 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:23.825729    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ad4893/auth-test-model-ad4893-kserve-7666559899-z8t85"]
Jul 17 11:50:24.006536 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:24.006503    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b842528-e00f-464c-8dab-81400143fec7-tls-certs\") pod \"1b842528-e00f-464c-8dab-81400143fec7\" (UID: \"1b842528-e00f-464c-8dab-81400143fec7\") "
Jul 17 11:50:24.006938 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:24.006552    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b842528-e00f-464c-8dab-81400143fec7-tmp-dir\") pod \"1b842528-e00f-464c-8dab-81400143fec7\" (UID: \"1b842528-e00f-464c-8dab-81400143fec7\") "
Jul 17 11:50:24.006938 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:24.006595    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b842528-e00f-464c-8dab-81400143fec7-home\") pod \"1b842528-e00f-464c-8dab-81400143fec7\" (UID: \"1b842528-e00f-464c-8dab-81400143fec7\") "
Jul 17 11:50:24.006938 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:24.006626    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2gvc\" (UniqueName: \"kubernetes.io/projected/1b842528-e00f-464c-8dab-81400143fec7-kube-api-access-n2gvc\") pod \"1b842528-e00f-464c-8dab-81400143fec7\" (UID: \"1b842528-e00f-464c-8dab-81400143fec7\") "
Jul 17 11:50:24.006938 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:24.006680    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b842528-e00f-464c-8dab-81400143fec7-dshm\") pod \"1b842528-e00f-464c-8dab-81400143fec7\" (UID: \"1b842528-e00f-464c-8dab-81400143fec7\") "
Jul 17 11:50:24.006938 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:24.006739    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b842528-e00f-464c-8dab-81400143fec7-model-cache\") pod \"1b842528-e00f-464c-8dab-81400143fec7\" (UID: \"1b842528-e00f-464c-8dab-81400143fec7\") "
Jul 17 11:50:24.006938 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:24.006886    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b842528-e00f-464c-8dab-81400143fec7-home" (OuterVolumeSpecName: "home") pod "1b842528-e00f-464c-8dab-81400143fec7" (UID: "1b842528-e00f-464c-8dab-81400143fec7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:50:24.006938 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:24.006917    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b842528-e00f-464c-8dab-81400143fec7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b842528-e00f-464c-8dab-81400143fec7" (UID: "1b842528-e00f-464c-8dab-81400143fec7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:50:24.007271 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:24.007055    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b842528-e00f-464c-8dab-81400143fec7-home\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:50:24.007271 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:24.007101    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b842528-e00f-464c-8dab-81400143fec7-tmp-dir\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:50:24.007271 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:24.007050    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b842528-e00f-464c-8dab-81400143fec7-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b842528-e00f-464c-8dab-81400143fec7" (UID: "1b842528-e00f-464c-8dab-81400143fec7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:50:24.008825 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:24.008797    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b842528-e00f-464c-8dab-81400143fec7-kube-api-access-n2gvc" (OuterVolumeSpecName: "kube-api-access-n2gvc") pod "1b842528-e00f-464c-8dab-81400143fec7" (UID: "1b842528-e00f-464c-8dab-81400143fec7"). InnerVolumeSpecName "kube-api-access-n2gvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 11:50:24.008923 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:24.008820    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b842528-e00f-464c-8dab-81400143fec7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b842528-e00f-464c-8dab-81400143fec7" (UID: "1b842528-e00f-464c-8dab-81400143fec7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 11:50:24.008923 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:24.008808    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b842528-e00f-464c-8dab-81400143fec7-dshm" (OuterVolumeSpecName: "dshm") pod "1b842528-e00f-464c-8dab-81400143fec7" (UID: "1b842528-e00f-464c-8dab-81400143fec7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:50:24.108481 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:24.108398    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b842528-e00f-464c-8dab-81400143fec7-dshm\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:50:24.108481 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:24.108426    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b842528-e00f-464c-8dab-81400143fec7-model-cache\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:50:24.108481 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:24.108437    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b842528-e00f-464c-8dab-81400143fec7-tls-certs\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:50:24.108481 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:24.108445    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2gvc\" (UniqueName: \"kubernetes.io/projected/1b842528-e00f-464c-8dab-81400143fec7-kube-api-access-n2gvc\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:50:24.809115 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:24.809085    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"
Jul 17 11:50:24.809115 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:24.809095    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz" event={"ID":"1b842528-e00f-464c-8dab-81400143fec7","Type":"ContainerDied","Data":"485dc1eaba45f53c4a10621637e121497bf87680d1ad0e9e8b3a3e25f93f8947"}
Jul 17 11:50:24.809387 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:24.809171    2563 scope.go:117] "RemoveContainer" containerID="e15d47c493b9869b60d81407cb6d78cd68dd91f86ac209e93faa84bfa5662bb2"
Jul 17 11:50:24.832353 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:24.832321    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"]
Jul 17 11:50:24.836851 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:24.836823    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-9d836f/auth-test-model-9d836f-kserve-7cbf4795f9-zggjz"]
Jul 17 11:50:25.665086 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:25.665045    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b842528-e00f-464c-8dab-81400143fec7" path="/var/lib/kubelet/pods/1b842528-e00f-464c-8dab-81400143fec7/volumes"
Jul 17 11:50:25.665466 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:25.665452    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c28fe762-9355-42b7-a514-f809873833bb" path="/var/lib/kubelet/pods/c28fe762-9355-42b7-a514-f809873833bb/volumes"
Jul 17 11:50:38.214112 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:38.214063    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-wkghz"]
Jul 17 11:50:38.214516 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:38.214404    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c28fe762-9355-42b7-a514-f809873833bb" containerName="main"
Jul 17 11:50:38.214516 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:38.214414    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c28fe762-9355-42b7-a514-f809873833bb" containerName="main"
Jul 17 11:50:38.214516 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:38.214435    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b842528-e00f-464c-8dab-81400143fec7" containerName="main"
Jul 17 11:50:38.214516 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:38.214441    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b842528-e00f-464c-8dab-81400143fec7" containerName="main"
Jul 17 11:50:38.214516 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:38.214493    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b842528-e00f-464c-8dab-81400143fec7" containerName="main"
Jul 17 11:50:38.214516 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:38.214500    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c28fe762-9355-42b7-a514-f809873833bb" containerName="main"
Jul 17 11:50:38.217376 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:38.217348    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-wkghz"
Jul 17 11:50:38.219742 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:38.219720    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 11:50:38.219850 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:38.219761    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwv2h\""
Jul 17 11:50:38.224802 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:38.224765    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-wkghz"]
Jul 17 11:50:38.224996 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:38.224969    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2685c4c7-5318-40a5-95ac-c3f09692a0be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45989be5-wkghz\" (UID: \"2685c4c7-5318-40a5-95ac-c3f09692a0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-wkghz"
Jul 17 11:50:38.325457 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:38.325425    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2685c4c7-5318-40a5-95ac-c3f09692a0be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45989be5-wkghz\" (UID: \"2685c4c7-5318-40a5-95ac-c3f09692a0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-wkghz"
Jul 17 11:50:38.326047 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:38.326030    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2685c4c7-5318-40a5-95ac-c3f09692a0be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45989be5-wkghz\" (UID: \"2685c4c7-5318-40a5-95ac-c3f09692a0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-wkghz"
Jul 17 11:50:38.528826 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:38.528742    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-wkghz"
Jul 17 11:50:38.660637 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:38.660612    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-wkghz"]
Jul 17 11:50:38.662547 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:50:38.662520    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2685c4c7_5318_40a5_95ac_c3f09692a0be.slice/crio-ca6280cef6acbd11bb0ddf35557604db1c4d0078e2c1041f1045a3ae01efcbf7 WatchSource:0}: Error finding container ca6280cef6acbd11bb0ddf35557604db1c4d0078e2c1041f1045a3ae01efcbf7: Status 404 returned error can't find the container with id ca6280cef6acbd11bb0ddf35557604db1c4d0078e2c1041f1045a3ae01efcbf7
Jul 17 11:50:38.884698 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:38.884664    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-wkghz" event={"ID":"2685c4c7-5318-40a5-95ac-c3f09692a0be","Type":"ContainerStarted","Data":"ca6280cef6acbd11bb0ddf35557604db1c4d0078e2c1041f1045a3ae01efcbf7"}
Jul 17 11:50:39.891513 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:39.891480    2563 generic.go:356] "Generic (PLEG): container finished" podID="2685c4c7-5318-40a5-95ac-c3f09692a0be" containerID="0f18e7cd7d07071b9c42dbe8a96259005de152e51a41681262c65ffaa13f2c4a" exitCode=6
Jul 17 11:50:39.892009 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:39.891520    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-wkghz" event={"ID":"2685c4c7-5318-40a5-95ac-c3f09692a0be","Type":"ContainerDied","Data":"0f18e7cd7d07071b9c42dbe8a96259005de152e51a41681262c65ffaa13f2c4a"}
Jul 17 11:50:39.892009 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:39.891810    2563 scope.go:117] "RemoveContainer" containerID="0f18e7cd7d07071b9c42dbe8a96259005de152e51a41681262c65ffaa13f2c4a"
Jul 17 11:50:40.898540 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:40.898508    2563 generic.go:356] "Generic (PLEG): container finished" podID="2685c4c7-5318-40a5-95ac-c3f09692a0be" containerID="8629c908d567ba3321b553b5b927592f49fb1003ee314439f2ab4af0b3ed5c52" exitCode=6
Jul 17 11:50:40.898954 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:40.898588    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-wkghz" event={"ID":"2685c4c7-5318-40a5-95ac-c3f09692a0be","Type":"ContainerDied","Data":"8629c908d567ba3321b553b5b927592f49fb1003ee314439f2ab4af0b3ed5c52"}
Jul 17 11:50:40.898954 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:40.898646    2563 scope.go:117] "RemoveContainer" containerID="0f18e7cd7d07071b9c42dbe8a96259005de152e51a41681262c65ffaa13f2c4a"
Jul 17 11:50:40.898954 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:40.898893    2563 scope.go:117] "RemoveContainer" containerID="8629c908d567ba3321b553b5b927592f49fb1003ee314439f2ab4af0b3ed5c52"
Jul 17 11:50:40.899164 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:50:40.899144    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45989be5-wkghz_odh-ai-gateway-infra(2685c4c7-5318-40a5-95ac-c3f09692a0be)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-wkghz" podUID="2685c4c7-5318-40a5-95ac-c3f09692a0be"
Jul 17 11:50:41.905838 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:41.905811    2563 scope.go:117] "RemoveContainer" containerID="8629c908d567ba3321b553b5b927592f49fb1003ee314439f2ab4af0b3ed5c52"
Jul 17 11:50:41.906270 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:50:41.906049    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45989be5-wkghz_odh-ai-gateway-infra(2685c4c7-5318-40a5-95ac-c3f09692a0be)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-wkghz" podUID="2685c4c7-5318-40a5-95ac-c3f09692a0be"
Jul 17 11:50:54.660240 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.660201    2563 scope.go:117] "RemoveContainer" containerID="8629c908d567ba3321b553b5b927592f49fb1003ee314439f2ab4af0b3ed5c52"
Jul 17 11:50:54.700895 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.700869    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"]
Jul 17 11:50:54.705703 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.705684    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"
Jul 17 11:50:54.709532 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.709505    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9d836f\"/\"sub-test-model-9d836f-kserve-self-signed-certs\""
Jul 17 11:50:54.709532 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.709524    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9d836f\"/\"default-dockercfg-mrs27\""
Jul 17 11:50:54.709709 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.709552    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9d836f\"/\"openshift-service-ca.crt\""
Jul 17 11:50:54.709709 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.709604    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9d836f\"/\"kube-root-ca.crt\""
Jul 17 11:50:54.714809 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.714772    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:50:54.715547 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.715526    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"]
Jul 17 11:50:54.773692 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.773656    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-tmp-dir\") pod \"sub-test-model-9d836f-kserve-54467dd955-wdr46\" (UID: \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\") " pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"
Jul 17 11:50:54.773904 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.773715    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-tls-certs\") pod \"sub-test-model-9d836f-kserve-54467dd955-wdr46\" (UID: \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\") " pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"
Jul 17 11:50:54.773904 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.773807    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-dshm\") pod \"sub-test-model-9d836f-kserve-54467dd955-wdr46\" (UID: \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\") " pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"
Jul 17 11:50:54.773904 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.773892    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-home\") pod \"sub-test-model-9d836f-kserve-54467dd955-wdr46\" (UID: \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\") " pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"
Jul 17 11:50:54.774062 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.773947    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2kct\" (UniqueName: \"kubernetes.io/projected/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-kube-api-access-j2kct\") pod \"sub-test-model-9d836f-kserve-54467dd955-wdr46\" (UID: \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\") " pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"
Jul 17 11:50:54.774062 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.774035    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-model-cache\") pod \"sub-test-model-9d836f-kserve-54467dd955-wdr46\" (UID: \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\") " pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"
Jul 17 11:50:54.874807 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.874773    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-model-cache\") pod \"sub-test-model-9d836f-kserve-54467dd955-wdr46\" (UID: \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\") " pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"
Jul 17 11:50:54.874933 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.874840    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-tmp-dir\") pod \"sub-test-model-9d836f-kserve-54467dd955-wdr46\" (UID: \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\") " pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"
Jul 17 11:50:54.874933 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.874898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-tls-certs\") pod \"sub-test-model-9d836f-kserve-54467dd955-wdr46\" (UID: \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\") " pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"
Jul 17 11:50:54.875088 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.874950    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-dshm\") pod \"sub-test-model-9d836f-kserve-54467dd955-wdr46\" (UID: \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\") " pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"
Jul 17 11:50:54.875088 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.875034    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-home\") pod \"sub-test-model-9d836f-kserve-54467dd955-wdr46\" (UID: \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\") " pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"
Jul 17 11:50:54.875185 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.875090    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2kct\" (UniqueName: \"kubernetes.io/projected/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-kube-api-access-j2kct\") pod \"sub-test-model-9d836f-kserve-54467dd955-wdr46\" (UID: \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\") " pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"
Jul 17 11:50:54.875185 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.875154    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-tmp-dir\") pod \"sub-test-model-9d836f-kserve-54467dd955-wdr46\" (UID: \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\") " pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"
Jul 17 11:50:54.875338 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.875319    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-model-cache\") pod \"sub-test-model-9d836f-kserve-54467dd955-wdr46\" (UID: \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\") " pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"
Jul 17 11:50:54.875543 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.875519    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-home\") pod \"sub-test-model-9d836f-kserve-54467dd955-wdr46\" (UID: \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\") " pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"
Jul 17 11:50:54.877244 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.877224    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-dshm\") pod \"sub-test-model-9d836f-kserve-54467dd955-wdr46\" (UID: \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\") " pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"
Jul 17 11:50:54.877458 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.877442    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-tls-certs\") pod \"sub-test-model-9d836f-kserve-54467dd955-wdr46\" (UID: \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\") " pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"
Jul 17 11:50:54.886118 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.886095    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2kct\" (UniqueName: \"kubernetes.io/projected/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-kube-api-access-j2kct\") pod \"sub-test-model-9d836f-kserve-54467dd955-wdr46\" (UID: \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\") " pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"
Jul 17 11:50:54.974685 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.974603    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-wkghz" event={"ID":"2685c4c7-5318-40a5-95ac-c3f09692a0be","Type":"ContainerStarted","Data":"d21f6bf2d7d1e7dde4c3f405d588f4cf28512b4cf60e6b34472abf263d4a1d38"}
Jul 17 11:50:54.991243 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:54.991195    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-wkghz" podStartSLOduration=16.991177872 podStartE2EDuration="16.991177872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:50:54.990707932 +0000 UTC m=+3209.920247573" watchObservedRunningTime="2026-07-17 11:50:54.991177872 +0000 UTC m=+3209.920717511"
Jul 17 11:50:55.015867 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:55.015833    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"
Jul 17 11:50:55.156025 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:55.155998    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"]
Jul 17 11:50:55.157174 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:50:55.157144    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b7d6c7a_8a44_482f_b596_39cb5c2de43b.slice/crio-6f09208ac1776b30a9ea631f405f3a89ee6ae4dbc457ddc1e94a6e35f2b35d6b WatchSource:0}: Error finding container 6f09208ac1776b30a9ea631f405f3a89ee6ae4dbc457ddc1e94a6e35f2b35d6b: Status 404 returned error can't find the container with id 6f09208ac1776b30a9ea631f405f3a89ee6ae4dbc457ddc1e94a6e35f2b35d6b
Jul 17 11:50:55.981716 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:55.981619    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46" event={"ID":"8b7d6c7a-8a44-482f-b596-39cb5c2de43b","Type":"ContainerStarted","Data":"774fe1fee74226d24115a658a98bee539bc9f9725298349ff61877170d3c9834"}
Jul 17 11:50:55.981716 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:55.981668    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46" event={"ID":"8b7d6c7a-8a44-482f-b596-39cb5c2de43b","Type":"ContainerStarted","Data":"6f09208ac1776b30a9ea631f405f3a89ee6ae4dbc457ddc1e94a6e35f2b35d6b"}
Jul 17 11:50:55.983473 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:55.983445    2563 generic.go:356] "Generic (PLEG): container finished" podID="2685c4c7-5318-40a5-95ac-c3f09692a0be" containerID="d21f6bf2d7d1e7dde4c3f405d588f4cf28512b4cf60e6b34472abf263d4a1d38" exitCode=6
Jul 17 11:50:55.983595 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:55.983517    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-wkghz" event={"ID":"2685c4c7-5318-40a5-95ac-c3f09692a0be","Type":"ContainerDied","Data":"d21f6bf2d7d1e7dde4c3f405d588f4cf28512b4cf60e6b34472abf263d4a1d38"}
Jul 17 11:50:55.983595 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:55.983580    2563 scope.go:117] "RemoveContainer" containerID="8629c908d567ba3321b553b5b927592f49fb1003ee314439f2ab4af0b3ed5c52"
Jul 17 11:50:55.983833 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:55.983820    2563 scope.go:117] "RemoveContainer" containerID="d21f6bf2d7d1e7dde4c3f405d588f4cf28512b4cf60e6b34472abf263d4a1d38"
Jul 17 11:50:55.984050 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:50:55.984032    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45989be5-wkghz_odh-ai-gateway-infra(2685c4c7-5318-40a5-95ac-c3f09692a0be)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-wkghz" podUID="2685c4c7-5318-40a5-95ac-c3f09692a0be"
Jul 17 11:50:56.000879 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:56.000835    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46" podStartSLOduration=2.000822295 podStartE2EDuration="2.000822295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:50:55.999699017 +0000 UTC m=+3210.929238661" watchObservedRunningTime="2026-07-17 11:50:56.000822295 +0000 UTC m=+3210.930361935"
Jul 17 11:50:56.004930 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:56.004903    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-wkghz"]
Jul 17 11:50:57.125009 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:57.124979    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-wkghz"
Jul 17 11:50:57.197287 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:57.197248    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2685c4c7-5318-40a5-95ac-c3f09692a0be-maas-api-service-ca\") pod \"2685c4c7-5318-40a5-95ac-c3f09692a0be\" (UID: \"2685c4c7-5318-40a5-95ac-c3f09692a0be\") "
Jul 17 11:50:57.197644 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:57.197623    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2685c4c7-5318-40a5-95ac-c3f09692a0be-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2685c4c7-5318-40a5-95ac-c3f09692a0be" (UID: "2685c4c7-5318-40a5-95ac-c3f09692a0be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:50:57.298229 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:57.298143    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2685c4c7-5318-40a5-95ac-c3f09692a0be-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:50:57.996572 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:57.996487    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-wkghz"
Jul 17 11:50:57.996572 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:57.996522    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-wkghz" event={"ID":"2685c4c7-5318-40a5-95ac-c3f09692a0be","Type":"ContainerDied","Data":"ca6280cef6acbd11bb0ddf35557604db1c4d0078e2c1041f1045a3ae01efcbf7"}
Jul 17 11:50:57.996780 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:57.996581    2563 scope.go:117] "RemoveContainer" containerID="d21f6bf2d7d1e7dde4c3f405d588f4cf28512b4cf60e6b34472abf263d4a1d38"
Jul 17 11:50:58.016997 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:58.016966    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-wkghz"]
Jul 17 11:50:58.021897 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:58.021872    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-wkghz"]
Jul 17 11:50:59.665954 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:50:59.665899    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2685c4c7-5318-40a5-95ac-c3f09692a0be" path="/var/lib/kubelet/pods/2685c4c7-5318-40a5-95ac-c3f09692a0be/volumes"
Jul 17 11:51:03.210784 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:03.210739    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:51:05.016336 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:05.016294    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"
Jul 17 11:51:05.016750 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:05.016367    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"
Jul 17 11:51:05.028783 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:05.028754    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"
Jul 17 11:51:05.045446 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:05.045420    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"
Jul 17 11:51:13.322601 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:13.322564    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:51:13.389093 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:13.389047    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:51:36.055470 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:36.055434    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-kzhzl"]
Jul 17 11:51:36.055845 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:36.055775    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2685c4c7-5318-40a5-95ac-c3f09692a0be" containerName="revoke-keys"
Jul 17 11:51:36.055845 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:36.055787    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2685c4c7-5318-40a5-95ac-c3f09692a0be" containerName="revoke-keys"
Jul 17 11:51:36.055845 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:36.055801    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2685c4c7-5318-40a5-95ac-c3f09692a0be" containerName="revoke-keys"
Jul 17 11:51:36.055845 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:36.055806    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2685c4c7-5318-40a5-95ac-c3f09692a0be" containerName="revoke-keys"
Jul 17 11:51:36.055845 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:36.055825    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2685c4c7-5318-40a5-95ac-c3f09692a0be" containerName="revoke-keys"
Jul 17 11:51:36.055845 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:36.055830    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2685c4c7-5318-40a5-95ac-c3f09692a0be" containerName="revoke-keys"
Jul 17 11:51:36.056033 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:36.055885    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2685c4c7-5318-40a5-95ac-c3f09692a0be" containerName="revoke-keys"
Jul 17 11:51:36.056033 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:36.055892    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2685c4c7-5318-40a5-95ac-c3f09692a0be" containerName="revoke-keys"
Jul 17 11:51:36.058739 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:36.058723    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-kzhzl"
Jul 17 11:51:36.061342 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:36.061316    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwv2h\""
Jul 17 11:51:36.061342 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:36.061341    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 11:51:36.067229 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:36.067203    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-kzhzl"]
Jul 17 11:51:36.142682 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:36.142647    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a298c7-5d7c-46b5-916c-5ac1400e88e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-57741a73-kzhzl\" (UID: \"36a298c7-5d7c-46b5-916c-5ac1400e88e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-kzhzl"
Jul 17 11:51:36.243633 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:36.243605    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a298c7-5d7c-46b5-916c-5ac1400e88e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-57741a73-kzhzl\" (UID: \"36a298c7-5d7c-46b5-916c-5ac1400e88e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-kzhzl"
Jul 17 11:51:36.244279 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:36.244262    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a298c7-5d7c-46b5-916c-5ac1400e88e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-57741a73-kzhzl\" (UID: \"36a298c7-5d7c-46b5-916c-5ac1400e88e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-kzhzl"
Jul 17 11:51:36.370219 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:36.370151    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-kzhzl"
Jul 17 11:51:36.507532 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:36.507509    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-kzhzl"]
Jul 17 11:51:37.198472 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:37.198435    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-kzhzl" event={"ID":"36a298c7-5d7c-46b5-916c-5ac1400e88e9","Type":"ContainerStarted","Data":"d7dca1e2198017b1f8a020d723e4e6e0aaa79b06ea5646e170bd734601e981ce"}
Jul 17 11:51:37.198844 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:37.198489    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-kzhzl" event={"ID":"36a298c7-5d7c-46b5-916c-5ac1400e88e9","Type":"ContainerStarted","Data":"b40c73ad9a0e8efa2936b3c9b14999c8b7c6fcebddaa304bb8c659918acd883a"}
Jul 17 11:51:37.216638 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:37.216589    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-kzhzl" podStartSLOduration=1.216574236 podStartE2EDuration="1.216574236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:51:37.214095076 +0000 UTC m=+3252.143634730" watchObservedRunningTime="2026-07-17 11:51:37.216574236 +0000 UTC m=+3252.146113877"
Jul 17 11:51:38.204815 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:38.204781    2563 generic.go:356] "Generic (PLEG): container finished" podID="36a298c7-5d7c-46b5-916c-5ac1400e88e9" containerID="d7dca1e2198017b1f8a020d723e4e6e0aaa79b06ea5646e170bd734601e981ce" exitCode=6
Jul 17 11:51:38.205233 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:38.204834    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-kzhzl" event={"ID":"36a298c7-5d7c-46b5-916c-5ac1400e88e9","Type":"ContainerDied","Data":"d7dca1e2198017b1f8a020d723e4e6e0aaa79b06ea5646e170bd734601e981ce"}
Jul 17 11:51:38.205233 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:38.205021    2563 scope.go:117] "RemoveContainer" containerID="d7dca1e2198017b1f8a020d723e4e6e0aaa79b06ea5646e170bd734601e981ce"
Jul 17 11:51:39.211409 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:39.211377    2563 generic.go:356] "Generic (PLEG): container finished" podID="36a298c7-5d7c-46b5-916c-5ac1400e88e9" containerID="d5a773c8b3ce90ccdd3759aec37617cd4d8cb97a14f920e5e2c0bf51a7a83965" exitCode=6
Jul 17 11:51:39.211839 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:39.211458    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-kzhzl" event={"ID":"36a298c7-5d7c-46b5-916c-5ac1400e88e9","Type":"ContainerDied","Data":"d5a773c8b3ce90ccdd3759aec37617cd4d8cb97a14f920e5e2c0bf51a7a83965"}
Jul 17 11:51:39.211839 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:39.211522    2563 scope.go:117] "RemoveContainer" containerID="d7dca1e2198017b1f8a020d723e4e6e0aaa79b06ea5646e170bd734601e981ce"
Jul 17 11:51:39.211839 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:39.211698    2563 scope.go:117] "RemoveContainer" containerID="d5a773c8b3ce90ccdd3759aec37617cd4d8cb97a14f920e5e2c0bf51a7a83965"
Jul 17 11:51:39.211966 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:51:39.211919    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-57741a73-kzhzl_odh-ai-gateway-infra(36a298c7-5d7c-46b5-916c-5ac1400e88e9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-kzhzl" podUID="36a298c7-5d7c-46b5-916c-5ac1400e88e9"
Jul 17 11:51:40.218671 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:40.218640    2563 scope.go:117] "RemoveContainer" containerID="d5a773c8b3ce90ccdd3759aec37617cd4d8cb97a14f920e5e2c0bf51a7a83965"
Jul 17 11:51:40.219059 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:51:40.218816    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-57741a73-kzhzl_odh-ai-gateway-infra(36a298c7-5d7c-46b5-916c-5ac1400e88e9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-kzhzl" podUID="36a298c7-5d7c-46b5-916c-5ac1400e88e9"
Jul 17 11:51:40.535236 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:40.535140    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:51:41.402555 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:41.402524    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"]
Jul 17 11:51:41.422211 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:41.422166    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:51:45.424138 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:45.424106    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:51:50.759754 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:50.759713    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:51:50.817037 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:50.817003    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:51:54.196573 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:54.196529    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:51:54.290769 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:54.290728    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:51:55.662689 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:55.662658    2563 scope.go:117] "RemoveContainer" containerID="d5a773c8b3ce90ccdd3759aec37617cd4d8cb97a14f920e5e2c0bf51a7a83965"
Jul 17 11:51:56.301720 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:56.301629    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-kzhzl" event={"ID":"36a298c7-5d7c-46b5-916c-5ac1400e88e9","Type":"ContainerStarted","Data":"36c61e76056312c6748fb032a82300007a8f98cf143a6b55a483adf71e7483ad"}
Jul 17 11:51:56.422968 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:56.422922    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46" podUID="8b7d6c7a-8a44-482f-b596-39cb5c2de43b" containerName="main" containerID="cri-o://774fe1fee74226d24115a658a98bee539bc9f9725298349ff61877170d3c9834" gracePeriod=45
Jul 17 11:51:56.668387 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:56.668363    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"
Jul 17 11:51:56.830814 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:56.830784    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-dshm\") pod \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\" (UID: \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\") "
Jul 17 11:51:56.830814 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:56.830822    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2kct\" (UniqueName: \"kubernetes.io/projected/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-kube-api-access-j2kct\") pod \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\" (UID: \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\") "
Jul 17 11:51:56.831018 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:56.830858    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-home\") pod \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\" (UID: \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\") "
Jul 17 11:51:56.831018 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:56.830967    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-model-cache\") pod \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\" (UID: \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\") "
Jul 17 11:51:56.831018 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:56.831012    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-tls-certs\") pod \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\" (UID: \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\") "
Jul 17 11:51:56.831216 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:56.831053    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-tmp-dir\") pod \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\" (UID: \"8b7d6c7a-8a44-482f-b596-39cb5c2de43b\") "
Jul 17 11:51:56.831216 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:56.831129    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-home" (OuterVolumeSpecName: "home") pod "8b7d6c7a-8a44-482f-b596-39cb5c2de43b" (UID: "8b7d6c7a-8a44-482f-b596-39cb5c2de43b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:51:56.831325 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:56.831268    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-model-cache" (OuterVolumeSpecName: "model-cache") pod "8b7d6c7a-8a44-482f-b596-39cb5c2de43b" (UID: "8b7d6c7a-8a44-482f-b596-39cb5c2de43b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:51:56.831373 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:56.831324    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8b7d6c7a-8a44-482f-b596-39cb5c2de43b" (UID: "8b7d6c7a-8a44-482f-b596-39cb5c2de43b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:51:56.831427 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:56.831392    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-model-cache\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:51:56.831427 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:56.831406    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-tmp-dir\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:51:56.831427 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:56.831416    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-home\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:51:56.833140 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:56.833119    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b7d6c7a-8a44-482f-b596-39cb5c2de43b" (UID: "8b7d6c7a-8a44-482f-b596-39cb5c2de43b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 11:51:56.833435 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:56.833410    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-kube-api-access-j2kct" (OuterVolumeSpecName: "kube-api-access-j2kct") pod "8b7d6c7a-8a44-482f-b596-39cb5c2de43b" (UID: "8b7d6c7a-8a44-482f-b596-39cb5c2de43b"). InnerVolumeSpecName "kube-api-access-j2kct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 11:51:56.833435 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:56.833411    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-dshm" (OuterVolumeSpecName: "dshm") pod "8b7d6c7a-8a44-482f-b596-39cb5c2de43b" (UID: "8b7d6c7a-8a44-482f-b596-39cb5c2de43b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:51:56.931869 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:56.931832    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-tls-certs\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:51:56.931869 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:56.931863    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-dshm\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:51:56.931869 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:56.931872    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2kct\" (UniqueName: \"kubernetes.io/projected/8b7d6c7a-8a44-482f-b596-39cb5c2de43b-kube-api-access-j2kct\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:51:57.308887 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:57.308773    2563 generic.go:356] "Generic (PLEG): container finished" podID="36a298c7-5d7c-46b5-916c-5ac1400e88e9" containerID="36c61e76056312c6748fb032a82300007a8f98cf143a6b55a483adf71e7483ad" exitCode=6
Jul 17 11:51:57.308887 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:57.308857    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-kzhzl" event={"ID":"36a298c7-5d7c-46b5-916c-5ac1400e88e9","Type":"ContainerDied","Data":"36c61e76056312c6748fb032a82300007a8f98cf143a6b55a483adf71e7483ad"}
Jul 17 11:51:57.309150 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:57.308916    2563 scope.go:117] "RemoveContainer" containerID="d5a773c8b3ce90ccdd3759aec37617cd4d8cb97a14f920e5e2c0bf51a7a83965"
Jul 17 11:51:57.309224 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:57.309209    2563 scope.go:117] "RemoveContainer" containerID="36c61e76056312c6748fb032a82300007a8f98cf143a6b55a483adf71e7483ad"
Jul 17 11:51:57.309476 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:51:57.309454    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-57741a73-kzhzl_odh-ai-gateway-infra(36a298c7-5d7c-46b5-916c-5ac1400e88e9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-kzhzl" podUID="36a298c7-5d7c-46b5-916c-5ac1400e88e9"
Jul 17 11:51:57.310366 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:57.310345    2563 generic.go:356] "Generic (PLEG): container finished" podID="8b7d6c7a-8a44-482f-b596-39cb5c2de43b" containerID="774fe1fee74226d24115a658a98bee539bc9f9725298349ff61877170d3c9834" exitCode=0
Jul 17 11:51:57.310437 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:57.310378    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"
Jul 17 11:51:57.310437 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:57.310389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46" event={"ID":"8b7d6c7a-8a44-482f-b596-39cb5c2de43b","Type":"ContainerDied","Data":"774fe1fee74226d24115a658a98bee539bc9f9725298349ff61877170d3c9834"}
Jul 17 11:51:57.310437 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:57.310411    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46" event={"ID":"8b7d6c7a-8a44-482f-b596-39cb5c2de43b","Type":"ContainerDied","Data":"6f09208ac1776b30a9ea631f405f3a89ee6ae4dbc457ddc1e94a6e35f2b35d6b"}
Jul 17 11:51:57.323518 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:57.323497    2563 scope.go:117] "RemoveContainer" containerID="774fe1fee74226d24115a658a98bee539bc9f9725298349ff61877170d3c9834"
Jul 17 11:51:57.332095 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:57.332049    2563 scope.go:117] "RemoveContainer" containerID="774fe1fee74226d24115a658a98bee539bc9f9725298349ff61877170d3c9834"
Jul 17 11:51:57.332351 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:51:57.332332    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"774fe1fee74226d24115a658a98bee539bc9f9725298349ff61877170d3c9834\": container with ID starting with 774fe1fee74226d24115a658a98bee539bc9f9725298349ff61877170d3c9834 not found: ID does not exist" containerID="774fe1fee74226d24115a658a98bee539bc9f9725298349ff61877170d3c9834"
Jul 17 11:51:57.332402 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:57.332359    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"774fe1fee74226d24115a658a98bee539bc9f9725298349ff61877170d3c9834"} err="failed to get container status \"774fe1fee74226d24115a658a98bee539bc9f9725298349ff61877170d3c9834\": rpc error: code = NotFound desc = could not find container \"774fe1fee74226d24115a658a98bee539bc9f9725298349ff61877170d3c9834\": container with ID starting with 774fe1fee74226d24115a658a98bee539bc9f9725298349ff61877170d3c9834 not found: ID does not exist"
Jul 17 11:51:57.338351 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:57.338329    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-kzhzl"]
Jul 17 11:51:57.343402 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:57.343383    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"]
Jul 17 11:51:57.349483 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:57.349466    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-9d836f/sub-test-model-9d836f-kserve-54467dd955-wdr46"]
Jul 17 11:51:57.665670 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:57.665633    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b7d6c7a-8a44-482f-b596-39cb5c2de43b" path="/var/lib/kubelet/pods/8b7d6c7a-8a44-482f-b596-39cb5c2de43b/volumes"
Jul 17 11:51:58.445119 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:58.445059    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-kzhzl"
Jul 17 11:51:58.546910 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:58.546880    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a298c7-5d7c-46b5-916c-5ac1400e88e9-maas-api-service-ca\") pod \"36a298c7-5d7c-46b5-916c-5ac1400e88e9\" (UID: \"36a298c7-5d7c-46b5-916c-5ac1400e88e9\") "
Jul 17 11:51:58.547299 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:58.547275    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36a298c7-5d7c-46b5-916c-5ac1400e88e9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "36a298c7-5d7c-46b5-916c-5ac1400e88e9" (UID: "36a298c7-5d7c-46b5-916c-5ac1400e88e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:51:58.647796 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:58.647768    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a298c7-5d7c-46b5-916c-5ac1400e88e9-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:51:59.326009 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:59.325981    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-kzhzl"
Jul 17 11:51:59.326009 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:59.325987    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-kzhzl" event={"ID":"36a298c7-5d7c-46b5-916c-5ac1400e88e9","Type":"ContainerDied","Data":"b40c73ad9a0e8efa2936b3c9b14999c8b7c6fcebddaa304bb8c659918acd883a"}
Jul 17 11:51:59.326282 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:59.326046    2563 scope.go:117] "RemoveContainer" containerID="36c61e76056312c6748fb032a82300007a8f98cf143a6b55a483adf71e7483ad"
Jul 17 11:51:59.349341 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:59.349320    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-kzhzl"]
Jul 17 11:51:59.355454 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:59.355433    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57741a73-kzhzl"]
Jul 17 11:51:59.666131 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:51:59.666100    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36a298c7-5d7c-46b5-916c-5ac1400e88e9" path="/var/lib/kubelet/pods/36a298c7-5d7c-46b5-916c-5ac1400e88e9/volumes"
Jul 17 11:52:09.201433 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.201401    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"]
Jul 17 11:52:09.201906 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.201887    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36a298c7-5d7c-46b5-916c-5ac1400e88e9" containerName="revoke-keys"
Jul 17 11:52:09.202032 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.201909    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="36a298c7-5d7c-46b5-916c-5ac1400e88e9" containerName="revoke-keys"
Jul 17 11:52:09.202032 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.201937    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b7d6c7a-8a44-482f-b596-39cb5c2de43b" containerName="main"
Jul 17 11:52:09.202032 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.201945    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b7d6c7a-8a44-482f-b596-39cb5c2de43b" containerName="main"
Jul 17 11:52:09.202032 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.201986    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36a298c7-5d7c-46b5-916c-5ac1400e88e9" containerName="revoke-keys"
Jul 17 11:52:09.202032 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.201995    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="36a298c7-5d7c-46b5-916c-5ac1400e88e9" containerName="revoke-keys"
Jul 17 11:52:09.202306 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.202099    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2685c4c7-5318-40a5-95ac-c3f09692a0be" containerName="revoke-keys"
Jul 17 11:52:09.202306 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.202114    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="36a298c7-5d7c-46b5-916c-5ac1400e88e9" containerName="revoke-keys"
Jul 17 11:52:09.202306 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.202123    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="36a298c7-5d7c-46b5-916c-5ac1400e88e9" containerName="revoke-keys"
Jul 17 11:52:09.202306 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.202134    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="36a298c7-5d7c-46b5-916c-5ac1400e88e9" containerName="revoke-keys"
Jul 17 11:52:09.202306 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.202146    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b7d6c7a-8a44-482f-b596-39cb5c2de43b" containerName="main"
Jul 17 11:52:09.204970 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.204951    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"
Jul 17 11:52:09.207821 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.207799    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9d836f\"/\"rate-test-model-9d836f-kserve-self-signed-certs\""
Jul 17 11:52:09.208923 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.208904    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9d836f\"/\"kube-root-ca.crt\""
Jul 17 11:52:09.209032 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.208909    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9d836f\"/\"default-dockercfg-mrs27\""
Jul 17 11:52:09.209032 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.208963    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9d836f\"/\"openshift-service-ca.crt\""
Jul 17 11:52:09.218103 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.217256    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"]
Jul 17 11:52:09.234136 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.234110    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-dshm\") pod \"rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t\" (UID: \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\") " pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"
Jul 17 11:52:09.234254 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.234140    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksnlm\" (UniqueName: \"kubernetes.io/projected/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-kube-api-access-ksnlm\") pod \"rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t\" (UID: \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\") " pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"
Jul 17 11:52:09.234254 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.234173    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-tmp-dir\") pod \"rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t\" (UID: \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\") " pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"
Jul 17 11:52:09.234254 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.234238    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-tls-certs\") pod \"rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t\" (UID: \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\") " pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"
Jul 17 11:52:09.234376 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.234324    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-model-cache\") pod \"rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t\" (UID: \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\") " pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"
Jul 17 11:52:09.234376 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.234364    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-home\") pod \"rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t\" (UID: \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\") " pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"
Jul 17 11:52:09.239459 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.239436    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:52:09.334979 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.334951    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-model-cache\") pod \"rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t\" (UID: \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\") " pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"
Jul 17 11:52:09.335124 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.335004    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-home\") pod \"rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t\" (UID: \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\") " pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"
Jul 17 11:52:09.335124 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.335040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-dshm\") pod \"rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t\" (UID: \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\") " pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"
Jul 17 11:52:09.335124 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.335058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksnlm\" (UniqueName: \"kubernetes.io/projected/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-kube-api-access-ksnlm\") pod \"rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t\" (UID: \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\") " pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"
Jul 17 11:52:09.335124 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.335112    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-tmp-dir\") pod \"rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t\" (UID: \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\") " pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"
Jul 17 11:52:09.335325 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.335213    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-tls-certs\") pod \"rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t\" (UID: \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\") " pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"
Jul 17 11:52:09.335383 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.335359    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-home\") pod \"rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t\" (UID: \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\") " pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"
Jul 17 11:52:09.335440 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.335422    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-model-cache\") pod \"rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t\" (UID: \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\") " pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"
Jul 17 11:52:09.335561 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.335528    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-tmp-dir\") pod \"rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t\" (UID: \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\") " pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"
Jul 17 11:52:09.337241 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.337220    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-dshm\") pod \"rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t\" (UID: \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\") " pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"
Jul 17 11:52:09.337530 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.337512    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-tls-certs\") pod \"rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t\" (UID: \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\") " pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"
Jul 17 11:52:09.346392 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.346373    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksnlm\" (UniqueName: \"kubernetes.io/projected/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-kube-api-access-ksnlm\") pod \"rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t\" (UID: \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\") " pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"
Jul 17 11:52:09.514681 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.514591    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"
Jul 17 11:52:09.651394 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:09.651365    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"]
Jul 17 11:52:09.652134 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:52:09.652108    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b7ff55a_8cdb_4c03_9099_2e44f8b1f911.slice/crio-9c5e067644356730d4fd03c2cb4c620cdcc640ae4a6add649b73bbfe7015ee6c WatchSource:0}: Error finding container 9c5e067644356730d4fd03c2cb4c620cdcc640ae4a6add649b73bbfe7015ee6c: Status 404 returned error can't find the container with id 9c5e067644356730d4fd03c2cb4c620cdcc640ae4a6add649b73bbfe7015ee6c
Jul 17 11:52:10.386827 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:10.386797    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t" event={"ID":"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911","Type":"ContainerStarted","Data":"047f72ffd8c5a157b65b5e7b53cc459db5c68d613975963c6c4050bcf8780075"}
Jul 17 11:52:10.386827 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:10.386835    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t" event={"ID":"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911","Type":"ContainerStarted","Data":"9c5e067644356730d4fd03c2cb4c620cdcc640ae4a6add649b73bbfe7015ee6c"}
Jul 17 11:52:10.406466 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:10.406424    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t" podStartSLOduration=1.406411667 podStartE2EDuration="1.406411667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:52:10.403557318 +0000 UTC m=+3285.333096974" watchObservedRunningTime="2026-07-17 11:52:10.406411667 +0000 UTC m=+3285.335951307"
Jul 17 11:52:19.515021 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:19.514989    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"
Jul 17 11:52:19.515522 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:19.515034    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"
Jul 17 11:52:19.527159 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:19.527132    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"
Jul 17 11:52:20.454454 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:20.454423    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"
Jul 17 11:52:20.528616 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:20.528581    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:52:25.848220 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:25.848114    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nj5mj_55d1de37-8635-4de8-ab59-4ebfd0346a40/ovn-acl-logging/0.log"
Jul 17 11:52:25.856112 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:25.856095    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nj5mj_55d1de37-8635-4de8-ab59-4ebfd0346a40/ovn-acl-logging/0.log"
Jul 17 11:52:33.277280 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:33.277228    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:52:35.549708 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:35.549678    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:52:35.649422 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:35.649394    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:52:43.146245 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:43.146208    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:52:43.244511 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:43.244472    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:52:52.464436 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:52.464357    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:52:58.131056 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:58.131020    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:52:58.976000 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:58.975968    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"]
Jul 17 11:52:59.003060 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:52:59.003028    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:53:00.011935 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:00.011895    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7659-139717857129648"]
Jul 17 11:53:00.012708 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:00.012673    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36a298c7-5d7c-46b5-916c-5ac1400e88e9" containerName="revoke-keys"
Jul 17 11:53:00.012708 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:00.012703    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="36a298c7-5d7c-46b5-916c-5ac1400e88e9" containerName="revoke-keys"
Jul 17 11:53:00.016265 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:00.016238    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7659-139717857129648"]
Jul 17 11:53:00.016463 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:00.016327    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7659-139717857129648"
Jul 17 11:53:00.019805 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:00.019785    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cvbgn\""
Jul 17 11:53:00.080279 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:00.080252    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzrf6\" (UniqueName: \"kubernetes.io/projected/665cedaa-d7a9-4c8c-bf35-1f4fe21dbab4-kube-api-access-mzrf6\") pod \"test-curl-7659-139717857129648\" (UID: \"665cedaa-d7a9-4c8c-bf35-1f4fe21dbab4\") " pod="opendatahub/test-curl-7659-139717857129648"
Jul 17 11:53:00.181155 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:00.181129    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzrf6\" (UniqueName: \"kubernetes.io/projected/665cedaa-d7a9-4c8c-bf35-1f4fe21dbab4-kube-api-access-mzrf6\") pod \"test-curl-7659-139717857129648\" (UID: \"665cedaa-d7a9-4c8c-bf35-1f4fe21dbab4\") " pod="opendatahub/test-curl-7659-139717857129648"
Jul 17 11:53:00.194607 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:00.194578    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzrf6\" (UniqueName: \"kubernetes.io/projected/665cedaa-d7a9-4c8c-bf35-1f4fe21dbab4-kube-api-access-mzrf6\") pod \"test-curl-7659-139717857129648\" (UID: \"665cedaa-d7a9-4c8c-bf35-1f4fe21dbab4\") " pod="opendatahub/test-curl-7659-139717857129648"
Jul 17 11:53:00.325846 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:00.325785    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7659-139717857129648"
Jul 17 11:53:00.458242 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:00.458217    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7659-139717857129648"]
Jul 17 11:53:00.460023 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:53:00.459992    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod665cedaa_d7a9_4c8c_bf35_1f4fe21dbab4.slice/crio-b62aaa559a5999adbb4f1ef62501df994c0eb91b0137b975c30a8c1826be2999 WatchSource:0}: Error finding container b62aaa559a5999adbb4f1ef62501df994c0eb91b0137b975c30a8c1826be2999: Status 404 returned error can't find the container with id b62aaa559a5999adbb4f1ef62501df994c0eb91b0137b975c30a8c1826be2999
Jul 17 11:53:00.461375 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:00.461352    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 11:53:00.650745 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:00.650707    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7659-139717857129648" event={"ID":"665cedaa-d7a9-4c8c-bf35-1f4fe21dbab4","Type":"ContainerStarted","Data":"b62aaa559a5999adbb4f1ef62501df994c0eb91b0137b975c30a8c1826be2999"}
Jul 17 11:53:01.656585 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:01.656549    2563 generic.go:356] "Generic (PLEG): container finished" podID="665cedaa-d7a9-4c8c-bf35-1f4fe21dbab4" containerID="2d10c1f02e8048b2044b22a0b037f9f277169a209f98e6a3406166a0f18681fb" exitCode=0
Jul 17 11:53:01.656971 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:01.656618    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7659-139717857129648" event={"ID":"665cedaa-d7a9-4c8c-bf35-1f4fe21dbab4","Type":"ContainerDied","Data":"2d10c1f02e8048b2044b22a0b037f9f277169a209f98e6a3406166a0f18681fb"}
Jul 17 11:53:02.787836 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:02.787809    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7659-139717857129648"
Jul 17 11:53:02.809404 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:02.809377    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7659-139717857129648_665cedaa-d7a9-4c8c-bf35-1f4fe21dbab4/test-curl-7659-139717857129648/0.log"
Jul 17 11:53:02.834178 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:02.834153    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7659-139717857129648"]
Jul 17 11:53:02.838127 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:02.838105    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7659-139717857129648"]
Jul 17 11:53:02.905785 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:02.905758    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzrf6\" (UniqueName: \"kubernetes.io/projected/665cedaa-d7a9-4c8c-bf35-1f4fe21dbab4-kube-api-access-mzrf6\") pod \"665cedaa-d7a9-4c8c-bf35-1f4fe21dbab4\" (UID: \"665cedaa-d7a9-4c8c-bf35-1f4fe21dbab4\") "
Jul 17 11:53:02.907887 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:02.907861    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/665cedaa-d7a9-4c8c-bf35-1f4fe21dbab4-kube-api-access-mzrf6" (OuterVolumeSpecName: "kube-api-access-mzrf6") pod "665cedaa-d7a9-4c8c-bf35-1f4fe21dbab4" (UID: "665cedaa-d7a9-4c8c-bf35-1f4fe21dbab4"). InnerVolumeSpecName "kube-api-access-mzrf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 11:53:03.006711 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:03.006675    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzrf6\" (UniqueName: \"kubernetes.io/projected/665cedaa-d7a9-4c8c-bf35-1f4fe21dbab4-kube-api-access-mzrf6\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:53:03.665746 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:03.665700    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="665cedaa-d7a9-4c8c-bf35-1f4fe21dbab4" path="/var/lib/kubelet/pods/665cedaa-d7a9-4c8c-bf35-1f4fe21dbab4/volumes"
Jul 17 11:53:03.668460 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:03.668438    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7659-139717857129648"
Jul 17 11:53:03.668599 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:03.668462    2563 scope.go:117] "RemoveContainer" containerID="2d10c1f02e8048b2044b22a0b037f9f277169a209f98e6a3406166a0f18681fb"
Jul 17 11:53:12.198195 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:12.198148    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7659-139717857094656"]
Jul 17 11:53:12.198623 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:12.198490    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="665cedaa-d7a9-4c8c-bf35-1f4fe21dbab4" containerName="test-curl-7659-139717857129648"
Jul 17 11:53:12.198623 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:12.198502    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="665cedaa-d7a9-4c8c-bf35-1f4fe21dbab4" containerName="test-curl-7659-139717857129648"
Jul 17 11:53:12.198623 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:12.198575    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="665cedaa-d7a9-4c8c-bf35-1f4fe21dbab4" containerName="test-curl-7659-139717857129648"
Jul 17 11:53:12.201596 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:12.201578    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7659-139717857094656"
Jul 17 11:53:12.204186 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:12.204167    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cvbgn\""
Jul 17 11:53:12.209119 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:12.208932    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7659-139717857094656"]
Jul 17 11:53:12.296768 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:12.296734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hwqj\" (UniqueName: \"kubernetes.io/projected/9338d5f5-1e2d-49a1-aff1-2912baffa59b-kube-api-access-5hwqj\") pod \"test-curl-7659-139717857094656\" (UID: \"9338d5f5-1e2d-49a1-aff1-2912baffa59b\") " pod="opendatahub/test-curl-7659-139717857094656"
Jul 17 11:53:12.398255 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:12.398216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hwqj\" (UniqueName: \"kubernetes.io/projected/9338d5f5-1e2d-49a1-aff1-2912baffa59b-kube-api-access-5hwqj\") pod \"test-curl-7659-139717857094656\" (UID: \"9338d5f5-1e2d-49a1-aff1-2912baffa59b\") " pod="opendatahub/test-curl-7659-139717857094656"
Jul 17 11:53:12.407265 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:12.407234    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hwqj\" (UniqueName: \"kubernetes.io/projected/9338d5f5-1e2d-49a1-aff1-2912baffa59b-kube-api-access-5hwqj\") pod \"test-curl-7659-139717857094656\" (UID: \"9338d5f5-1e2d-49a1-aff1-2912baffa59b\") " pod="opendatahub/test-curl-7659-139717857094656"
Jul 17 11:53:12.511396 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:12.511297    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7659-139717857094656"
Jul 17 11:53:12.648302 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:12.648273    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7659-139717857094656"]
Jul 17 11:53:12.649949 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:53:12.649918    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9338d5f5_1e2d_49a1_aff1_2912baffa59b.slice/crio-fbf638919be5f773a31117101a4704c857e1576bd0f6224c376560d6a7ef2980 WatchSource:0}: Error finding container fbf638919be5f773a31117101a4704c857e1576bd0f6224c376560d6a7ef2980: Status 404 returned error can't find the container with id fbf638919be5f773a31117101a4704c857e1576bd0f6224c376560d6a7ef2980
Jul 17 11:53:12.713968 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:12.713934    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7659-139717857094656" event={"ID":"9338d5f5-1e2d-49a1-aff1-2912baffa59b","Type":"ContainerStarted","Data":"fbf638919be5f773a31117101a4704c857e1576bd0f6224c376560d6a7ef2980"}
Jul 17 11:53:13.720466 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:13.720375    2563 generic.go:356] "Generic (PLEG): container finished" podID="9338d5f5-1e2d-49a1-aff1-2912baffa59b" containerID="f750ef29b94d2e4608efd2b8329fa5909a2a4cf8370eec9415eaf2a7073663ad" exitCode=0
Jul 17 11:53:13.720842 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:13.720461    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7659-139717857094656" event={"ID":"9338d5f5-1e2d-49a1-aff1-2912baffa59b","Type":"ContainerDied","Data":"f750ef29b94d2e4608efd2b8329fa5909a2a4cf8370eec9415eaf2a7073663ad"}
Jul 17 11:53:13.995536 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:13.995436    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t" podUID="1b7ff55a-8cdb-4c03-9099-2e44f8b1f911" containerName="main" containerID="cri-o://047f72ffd8c5a157b65b5e7b53cc459db5c68d613975963c6c4050bcf8780075" gracePeriod=45
Jul 17 11:53:14.248867 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.248805    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"
Jul 17 11:53:14.316796 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.316762    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-tmp-dir\") pod \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\" (UID: \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\") "
Jul 17 11:53:14.316979 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.316806    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-dshm\") pod \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\" (UID: \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\") "
Jul 17 11:53:14.316979 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.316846    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-tls-certs\") pod \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\" (UID: \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\") "
Jul 17 11:53:14.316979 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.316931    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-model-cache\") pod \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\" (UID: \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\") "
Jul 17 11:53:14.316979 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.316967    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksnlm\" (UniqueName: \"kubernetes.io/projected/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-kube-api-access-ksnlm\") pod \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\" (UID: \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\") "
Jul 17 11:53:14.317192 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.316996    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-home\") pod \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\" (UID: \"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911\") "
Jul 17 11:53:14.317192 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.317115    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b7ff55a-8cdb-4c03-9099-2e44f8b1f911" (UID: "1b7ff55a-8cdb-4c03-9099-2e44f8b1f911"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:53:14.317275 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.317220    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b7ff55a-8cdb-4c03-9099-2e44f8b1f911" (UID: "1b7ff55a-8cdb-4c03-9099-2e44f8b1f911"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:53:14.317339 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.317329    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-model-cache\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:53:14.317391 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.317330    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-home" (OuterVolumeSpecName: "home") pod "1b7ff55a-8cdb-4c03-9099-2e44f8b1f911" (UID: "1b7ff55a-8cdb-4c03-9099-2e44f8b1f911"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:53:14.317391 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.317348    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-tmp-dir\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:53:14.318917 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.318889    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-dshm" (OuterVolumeSpecName: "dshm") pod "1b7ff55a-8cdb-4c03-9099-2e44f8b1f911" (UID: "1b7ff55a-8cdb-4c03-9099-2e44f8b1f911"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:53:14.319097 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.319049    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b7ff55a-8cdb-4c03-9099-2e44f8b1f911" (UID: "1b7ff55a-8cdb-4c03-9099-2e44f8b1f911"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 11:53:14.319220 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.319094    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-kube-api-access-ksnlm" (OuterVolumeSpecName: "kube-api-access-ksnlm") pod "1b7ff55a-8cdb-4c03-9099-2e44f8b1f911" (UID: "1b7ff55a-8cdb-4c03-9099-2e44f8b1f911"). InnerVolumeSpecName "kube-api-access-ksnlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 11:53:14.418164 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.418132    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-dshm\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:53:14.418164 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.418165    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-tls-certs\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:53:14.418417 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.418178    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksnlm\" (UniqueName: \"kubernetes.io/projected/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-kube-api-access-ksnlm\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:53:14.418417 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.418191    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911-home\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:53:14.726848 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.726810    2563 generic.go:356] "Generic (PLEG): container finished" podID="1b7ff55a-8cdb-4c03-9099-2e44f8b1f911" containerID="047f72ffd8c5a157b65b5e7b53cc459db5c68d613975963c6c4050bcf8780075" exitCode=0
Jul 17 11:53:14.726848 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.726849    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"
Jul 17 11:53:14.727416 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.726888    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t" event={"ID":"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911","Type":"ContainerDied","Data":"047f72ffd8c5a157b65b5e7b53cc459db5c68d613975963c6c4050bcf8780075"}
Jul 17 11:53:14.727416 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.726926    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t" event={"ID":"1b7ff55a-8cdb-4c03-9099-2e44f8b1f911","Type":"ContainerDied","Data":"9c5e067644356730d4fd03c2cb4c620cdcc640ae4a6add649b73bbfe7015ee6c"}
Jul 17 11:53:14.727416 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.726945    2563 scope.go:117] "RemoveContainer" containerID="047f72ffd8c5a157b65b5e7b53cc459db5c68d613975963c6c4050bcf8780075"
Jul 17 11:53:14.736708 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.736630    2563 scope.go:117] "RemoveContainer" containerID="047f72ffd8c5a157b65b5e7b53cc459db5c68d613975963c6c4050bcf8780075"
Jul 17 11:53:14.737008 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:53:14.736984    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"047f72ffd8c5a157b65b5e7b53cc459db5c68d613975963c6c4050bcf8780075\": container with ID starting with 047f72ffd8c5a157b65b5e7b53cc459db5c68d613975963c6c4050bcf8780075 not found: ID does not exist" containerID="047f72ffd8c5a157b65b5e7b53cc459db5c68d613975963c6c4050bcf8780075"
Jul 17 11:53:14.737108 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.737017    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"047f72ffd8c5a157b65b5e7b53cc459db5c68d613975963c6c4050bcf8780075"} err="failed to get container status \"047f72ffd8c5a157b65b5e7b53cc459db5c68d613975963c6c4050bcf8780075\": rpc error: code = NotFound desc = could not find container \"047f72ffd8c5a157b65b5e7b53cc459db5c68d613975963c6c4050bcf8780075\": container with ID starting with 047f72ffd8c5a157b65b5e7b53cc459db5c68d613975963c6c4050bcf8780075 not found: ID does not exist"
Jul 17 11:53:14.752264 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.752234    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"]
Jul 17 11:53:14.756784 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.756764    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-9d836f/rate-test-model-9d836f-kserve-5cd7d74fc9-9w88t"]
Jul 17 11:53:14.864403 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.864377    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7659-139717857094656"
Jul 17 11:53:14.885778 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.885749    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7659-139717857094656_9338d5f5-1e2d-49a1-aff1-2912baffa59b/test-curl-7659-139717857094656/0.log"
Jul 17 11:53:14.908714 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.908691    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7659-139717857094656"]
Jul 17 11:53:14.912848 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.912828    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7659-139717857094656"]
Jul 17 11:53:14.922160 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.922145    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hwqj\" (UniqueName: \"kubernetes.io/projected/9338d5f5-1e2d-49a1-aff1-2912baffa59b-kube-api-access-5hwqj\") pod \"9338d5f5-1e2d-49a1-aff1-2912baffa59b\" (UID: \"9338d5f5-1e2d-49a1-aff1-2912baffa59b\") "
Jul 17 11:53:14.924090 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:14.924057    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9338d5f5-1e2d-49a1-aff1-2912baffa59b-kube-api-access-5hwqj" (OuterVolumeSpecName: "kube-api-access-5hwqj") pod "9338d5f5-1e2d-49a1-aff1-2912baffa59b" (UID: "9338d5f5-1e2d-49a1-aff1-2912baffa59b"). InnerVolumeSpecName "kube-api-access-5hwqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 11:53:15.023730 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:15.023658    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hwqj\" (UniqueName: \"kubernetes.io/projected/9338d5f5-1e2d-49a1-aff1-2912baffa59b-kube-api-access-5hwqj\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:53:15.038744 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:15.038715    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7659-139717857094656"]
Jul 17 11:53:15.039129 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:15.039112    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b7ff55a-8cdb-4c03-9099-2e44f8b1f911" containerName="main"
Jul 17 11:53:15.039240 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:15.039130    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b7ff55a-8cdb-4c03-9099-2e44f8b1f911" containerName="main"
Jul 17 11:53:15.039240 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:15.039144    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9338d5f5-1e2d-49a1-aff1-2912baffa59b" containerName="test-curl-7659-139717857094656"
Jul 17 11:53:15.039240 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:15.039153    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9338d5f5-1e2d-49a1-aff1-2912baffa59b" containerName="test-curl-7659-139717857094656"
Jul 17 11:53:15.039388 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:15.039282    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b7ff55a-8cdb-4c03-9099-2e44f8b1f911" containerName="main"
Jul 17 11:53:15.039388 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:15.039295    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9338d5f5-1e2d-49a1-aff1-2912baffa59b" containerName="test-curl-7659-139717857094656"
Jul 17 11:53:15.042417 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:15.042400    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7659-139717857094656"
Jul 17 11:53:15.048648 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:15.048620    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7659-139717857094656"]
Jul 17 11:53:15.124560 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:15.124532    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgs8w\" (UniqueName: \"kubernetes.io/projected/45313f98-6678-4a41-824f-f9fc7063b028-kube-api-access-hgs8w\") pod \"test-curl-7659-139717857094656\" (UID: \"45313f98-6678-4a41-824f-f9fc7063b028\") " pod="opendatahub/test-curl-7659-139717857094656"
Jul 17 11:53:15.225873 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:15.225848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgs8w\" (UniqueName: \"kubernetes.io/projected/45313f98-6678-4a41-824f-f9fc7063b028-kube-api-access-hgs8w\") pod \"test-curl-7659-139717857094656\" (UID: \"45313f98-6678-4a41-824f-f9fc7063b028\") " pod="opendatahub/test-curl-7659-139717857094656"
Jul 17 11:53:15.235488 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:15.235464    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgs8w\" (UniqueName: \"kubernetes.io/projected/45313f98-6678-4a41-824f-f9fc7063b028-kube-api-access-hgs8w\") pod \"test-curl-7659-139717857094656\" (UID: \"45313f98-6678-4a41-824f-f9fc7063b028\") " pod="opendatahub/test-curl-7659-139717857094656"
Jul 17 11:53:15.352287 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:15.352209    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7659-139717857094656"
Jul 17 11:53:15.478511 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:15.478484    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7659-139717857094656"]
Jul 17 11:53:15.480082 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:53:15.480045    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45313f98_6678_4a41_824f_f9fc7063b028.slice/crio-957dde3ce8289ca3db32ebc3c14b9ab8f1dd8cd8ea90e20edaa0f560d12d71e2 WatchSource:0}: Error finding container 957dde3ce8289ca3db32ebc3c14b9ab8f1dd8cd8ea90e20edaa0f560d12d71e2: Status 404 returned error can't find the container with id 957dde3ce8289ca3db32ebc3c14b9ab8f1dd8cd8ea90e20edaa0f560d12d71e2
Jul 17 11:53:15.668600 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:15.668573    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b7ff55a-8cdb-4c03-9099-2e44f8b1f911" path="/var/lib/kubelet/pods/1b7ff55a-8cdb-4c03-9099-2e44f8b1f911/volumes"
Jul 17 11:53:15.668901 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:15.668889    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9338d5f5-1e2d-49a1-aff1-2912baffa59b" path="/var/lib/kubelet/pods/9338d5f5-1e2d-49a1-aff1-2912baffa59b/volumes"
Jul 17 11:53:15.734882 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:15.734861    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7659-139717857094656"
Jul 17 11:53:15.735299 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:15.734863    2563 scope.go:117] "RemoveContainer" containerID="f750ef29b94d2e4608efd2b8329fa5909a2a4cf8370eec9415eaf2a7073663ad"
Jul 17 11:53:15.736036 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:15.736009    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7659-139717857094656" event={"ID":"45313f98-6678-4a41-824f-f9fc7063b028","Type":"ContainerStarted","Data":"957dde3ce8289ca3db32ebc3c14b9ab8f1dd8cd8ea90e20edaa0f560d12d71e2"}
Jul 17 11:53:16.743709 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:16.743674    2563 generic.go:356] "Generic (PLEG): container finished" podID="45313f98-6678-4a41-824f-f9fc7063b028" containerID="5d3ddeaedceb8773407221987d067f7b2c8ceb59eb072af4162463afadd7168c" exitCode=0
Jul 17 11:53:16.744180 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:16.743723    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7659-139717857094656" event={"ID":"45313f98-6678-4a41-824f-f9fc7063b028","Type":"ContainerDied","Data":"5d3ddeaedceb8773407221987d067f7b2c8ceb59eb072af4162463afadd7168c"}
Jul 17 11:53:17.875160 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:17.875138    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7659-139717857094656"
Jul 17 11:53:17.896419 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:17.896384    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7659-139717857094656_45313f98-6678-4a41-824f-f9fc7063b028/test-curl-7659-139717857094656/0.log"
Jul 17 11:53:17.921522 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:17.921494    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7659-139717857094656"]
Jul 17 11:53:17.927266 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:17.927244    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7659-139717857094656"]
Jul 17 11:53:17.950312 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:17.950290    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgs8w\" (UniqueName: \"kubernetes.io/projected/45313f98-6678-4a41-824f-f9fc7063b028-kube-api-access-hgs8w\") pod \"45313f98-6678-4a41-824f-f9fc7063b028\" (UID: \"45313f98-6678-4a41-824f-f9fc7063b028\") "
Jul 17 11:53:17.952236 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:17.952211    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45313f98-6678-4a41-824f-f9fc7063b028-kube-api-access-hgs8w" (OuterVolumeSpecName: "kube-api-access-hgs8w") pod "45313f98-6678-4a41-824f-f9fc7063b028" (UID: "45313f98-6678-4a41-824f-f9fc7063b028"). InnerVolumeSpecName "kube-api-access-hgs8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 11:53:18.051459 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:18.051394    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgs8w\" (UniqueName: \"kubernetes.io/projected/45313f98-6678-4a41-824f-f9fc7063b028-kube-api-access-hgs8w\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:53:18.754354 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:18.754328    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7659-139717857094656"
Jul 17 11:53:18.754526 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:18.754328    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="957dde3ce8289ca3db32ebc3c14b9ab8f1dd8cd8ea90e20edaa0f560d12d71e2"
Jul 17 11:53:18.764373 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:53:18.764344    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7659-139717857094656\" is forbidden: User \"system:node:ip-10-0-140-233.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-233.ec2.internal' and this object" podUID="45313f98-6678-4a41-824f-f9fc7063b028" pod="opendatahub/test-curl-7659-139717857094656"
Jul 17 11:53:19.621944 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:19.621896    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:53:19.665099 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:19.665056    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45313f98-6678-4a41-824f-f9fc7063b028" path="/var/lib/kubelet/pods/45313f98-6678-4a41-824f-f9fc7063b028/volumes"
Jul 17 11:53:19.694795 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:19.694764    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:53:28.637280 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:28.637246    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:53:33.226997 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:33.226960    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:53:39.644260 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:39.644218    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:53:39.743669 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:39.743625    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:53:43.980034 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:43.980004    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"]
Jul 17 11:53:43.980426 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:43.980228    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh" podUID="9c1b08e5-60d6-4fdc-83eb-d5365a9d0661" containerName="istio-proxy" containerID="cri-o://fe1da9c6f7a0707767bea652e22ad83b5f2721fea3a76422068da459b8796b17" gracePeriod=30
Jul 17 11:53:43.997557 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:43.997524    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:53:44.244001 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:44.243898    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:53:45.238811 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:45.238775    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:53:45.341675 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:45.341639    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:53:49.228186 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.228161    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:53:49.340689 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.340589    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istio-token\") pod \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") "
Jul 17 11:53:49.340689 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.340655    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-workload-socket\") pod \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") "
Jul 17 11:53:49.340689 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.340683    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6cj2\" (UniqueName: \"kubernetes.io/projected/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-kube-api-access-m6cj2\") pod \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") "
Jul 17 11:53:49.341030 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.340717    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-credential-socket\") pod \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") "
Jul 17 11:53:49.341030 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.340747    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istio-data\") pod \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") "
Jul 17 11:53:49.341030 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.340778    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-workload-certs\") pod \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") "
Jul 17 11:53:49.341030 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.340811    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istio-podinfo\") pod \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") "
Jul 17 11:53:49.341030 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.340863    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istiod-ca-cert\") pod \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") "
Jul 17 11:53:49.341030 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.340891    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istio-envoy\") pod \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\" (UID: \"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661\") "
Jul 17 11:53:49.341030 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.340954    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9c1b08e5-60d6-4fdc-83eb-d5365a9d0661" (UID: "9c1b08e5-60d6-4fdc-83eb-d5365a9d0661"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:53:49.341448 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.341220    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-workload-socket\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:53:49.341448 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.341317    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9c1b08e5-60d6-4fdc-83eb-d5365a9d0661" (UID: "9c1b08e5-60d6-4fdc-83eb-d5365a9d0661"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:53:49.341448 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.341396    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9c1b08e5-60d6-4fdc-83eb-d5365a9d0661" (UID: "9c1b08e5-60d6-4fdc-83eb-d5365a9d0661"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:53:49.341448 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.341441    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9c1b08e5-60d6-4fdc-83eb-d5365a9d0661" (UID: "9c1b08e5-60d6-4fdc-83eb-d5365a9d0661"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:53:49.341917 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.341877    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istio-data" (OuterVolumeSpecName: "istio-data") pod "9c1b08e5-60d6-4fdc-83eb-d5365a9d0661" (UID: "9c1b08e5-60d6-4fdc-83eb-d5365a9d0661"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:53:49.343245 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.343215    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9c1b08e5-60d6-4fdc-83eb-d5365a9d0661" (UID: "9c1b08e5-60d6-4fdc-83eb-d5365a9d0661"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 11:53:49.343245 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.343232    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9c1b08e5-60d6-4fdc-83eb-d5365a9d0661" (UID: "9c1b08e5-60d6-4fdc-83eb-d5365a9d0661"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 11:53:49.343384 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.343262    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-kube-api-access-m6cj2" (OuterVolumeSpecName: "kube-api-access-m6cj2") pod "9c1b08e5-60d6-4fdc-83eb-d5365a9d0661" (UID: "9c1b08e5-60d6-4fdc-83eb-d5365a9d0661"). InnerVolumeSpecName "kube-api-access-m6cj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 11:53:49.343530 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.343513    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istio-token" (OuterVolumeSpecName: "istio-token") pod "9c1b08e5-60d6-4fdc-83eb-d5365a9d0661" (UID: "9c1b08e5-60d6-4fdc-83eb-d5365a9d0661"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 11:53:49.441829 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.441792    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istiod-ca-cert\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:53:49.441829 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.441824    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istio-envoy\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:53:49.441829 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.441832    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istio-token\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:53:49.442118 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.441842    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6cj2\" (UniqueName: \"kubernetes.io/projected/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-kube-api-access-m6cj2\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:53:49.442118 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.441854    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-credential-socket\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:53:49.442118 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.441866    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istio-data\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:53:49.442118 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.441874    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-workload-certs\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:53:49.442118 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.441882    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661-istio-podinfo\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:53:49.915158 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.915123    2563 generic.go:356] "Generic (PLEG): container finished" podID="9c1b08e5-60d6-4fdc-83eb-d5365a9d0661" containerID="fe1da9c6f7a0707767bea652e22ad83b5f2721fea3a76422068da459b8796b17" exitCode=0
Jul 17 11:53:49.915330 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.915172    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"
Jul 17 11:53:49.915330 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.915204    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh" event={"ID":"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661","Type":"ContainerDied","Data":"fe1da9c6f7a0707767bea652e22ad83b5f2721fea3a76422068da459b8796b17"}
Jul 17 11:53:49.915330 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.915263    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh" event={"ID":"9c1b08e5-60d6-4fdc-83eb-d5365a9d0661","Type":"ContainerDied","Data":"68d019a891b92714c3e58ac4c036375764250992f7f8a25a3667c1995eab1dba"}
Jul 17 11:53:49.915330 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.915292    2563 scope.go:117] "RemoveContainer" containerID="fe1da9c6f7a0707767bea652e22ad83b5f2721fea3a76422068da459b8796b17"
Jul 17 11:53:49.924317 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.924285    2563 scope.go:117] "RemoveContainer" containerID="eaf5710e8454e1fbd311b9cee2ae2f1ddb15a7b751713e855ef408aaad35dac8"
Jul 17 11:53:49.932363 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.932345    2563 scope.go:117] "RemoveContainer" containerID="fe1da9c6f7a0707767bea652e22ad83b5f2721fea3a76422068da459b8796b17"
Jul 17 11:53:49.932630 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:53:49.932609    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe1da9c6f7a0707767bea652e22ad83b5f2721fea3a76422068da459b8796b17\": container with ID starting with fe1da9c6f7a0707767bea652e22ad83b5f2721fea3a76422068da459b8796b17 not found: ID does not exist" containerID="fe1da9c6f7a0707767bea652e22ad83b5f2721fea3a76422068da459b8796b17"
Jul 17 11:53:49.932715 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.932641    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe1da9c6f7a0707767bea652e22ad83b5f2721fea3a76422068da459b8796b17"} err="failed to get container status \"fe1da9c6f7a0707767bea652e22ad83b5f2721fea3a76422068da459b8796b17\": rpc error: code = NotFound desc = could not find container \"fe1da9c6f7a0707767bea652e22ad83b5f2721fea3a76422068da459b8796b17\": container with ID starting with fe1da9c6f7a0707767bea652e22ad83b5f2721fea3a76422068da459b8796b17 not found: ID does not exist"
Jul 17 11:53:49.932715 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.932665    2563 scope.go:117] "RemoveContainer" containerID="eaf5710e8454e1fbd311b9cee2ae2f1ddb15a7b751713e855ef408aaad35dac8"
Jul 17 11:53:49.932925 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:53:49.932905    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaf5710e8454e1fbd311b9cee2ae2f1ddb15a7b751713e855ef408aaad35dac8\": container with ID starting with eaf5710e8454e1fbd311b9cee2ae2f1ddb15a7b751713e855ef408aaad35dac8 not found: ID does not exist" containerID="eaf5710e8454e1fbd311b9cee2ae2f1ddb15a7b751713e855ef408aaad35dac8"
Jul 17 11:53:49.932965 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.932932    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaf5710e8454e1fbd311b9cee2ae2f1ddb15a7b751713e855ef408aaad35dac8"} err="failed to get container status \"eaf5710e8454e1fbd311b9cee2ae2f1ddb15a7b751713e855ef408aaad35dac8\": rpc error: code = NotFound desc = could not find container \"eaf5710e8454e1fbd311b9cee2ae2f1ddb15a7b751713e855ef408aaad35dac8\": container with ID starting with eaf5710e8454e1fbd311b9cee2ae2f1ddb15a7b751713e855ef408aaad35dac8 not found: ID does not exist"
Jul 17 11:53:49.938392 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.938366    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"]
Jul 17 11:53:49.945240 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:49.945216    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-ad4893-openshift-default-66d8fd856c-7cvdh"]
Jul 17 11:53:51.665422 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:51.665387    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c1b08e5-60d6-4fdc-83eb-d5365a9d0661" path="/var/lib/kubelet/pods/9c1b08e5-60d6-4fdc-83eb-d5365a9d0661/volumes"
Jul 17 11:53:54.915842 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:54.915805    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:53:57.440727 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:57.440695    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:53:57.537048 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:57.537001    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:53:59.489190 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:53:59.489155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:54:05.879867 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:05.879832    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:54:05.964742 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:05.964702    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:54:08.423354 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:08.423313    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-zt729"]
Jul 17 11:54:08.423828 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:08.423775    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c1b08e5-60d6-4fdc-83eb-d5365a9d0661" containerName="istio-proxy"
Jul 17 11:54:08.423828 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:08.423793    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c1b08e5-60d6-4fdc-83eb-d5365a9d0661" containerName="istio-proxy"
Jul 17 11:54:08.423828 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:08.423825    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45313f98-6678-4a41-824f-f9fc7063b028" containerName="test-curl-7659-139717857094656"
Jul 17 11:54:08.424009 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:08.423852    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="45313f98-6678-4a41-824f-f9fc7063b028" containerName="test-curl-7659-139717857094656"
Jul 17 11:54:08.424009 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:08.423947    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c1b08e5-60d6-4fdc-83eb-d5365a9d0661" containerName="istio-proxy"
Jul 17 11:54:08.424009 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:08.423961    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c1b08e5-60d6-4fdc-83eb-d5365a9d0661" containerName="istio-proxy"
Jul 17 11:54:08.424009 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:08.423972    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="45313f98-6678-4a41-824f-f9fc7063b028" containerName="test-curl-7659-139717857094656"
Jul 17 11:54:08.426938 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:08.426915    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-zt729"
Jul 17 11:54:08.429769 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:08.429745    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 11:54:08.429966 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:08.429952    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwv2h\""
Jul 17 11:54:08.436099 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:08.436056    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-zt729"]
Jul 17 11:54:08.512424 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:08.512380    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8639e56-1824-4140-ace7-402d112c87ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45989be5-zt729\" (UID: \"a8639e56-1824-4140-ace7-402d112c87ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-zt729"
Jul 17 11:54:08.613159 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:08.613127    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8639e56-1824-4140-ace7-402d112c87ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45989be5-zt729\" (UID: \"a8639e56-1824-4140-ace7-402d112c87ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-zt729"
Jul 17 11:54:08.613781 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:08.613759    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8639e56-1824-4140-ace7-402d112c87ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45989be5-zt729\" (UID: \"a8639e56-1824-4140-ace7-402d112c87ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-zt729"
Jul 17 11:54:08.738469 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:08.738386    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-zt729"
Jul 17 11:54:08.872134 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:08.872103    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-zt729"]
Jul 17 11:54:08.873438 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:54:08.873415    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8639e56_1824_4140_ace7_402d112c87ed.slice/crio-dc2387fa0512a0444f26abb8f63d8d2a43d5401da824a8f70433addb2f82e388 WatchSource:0}: Error finding container dc2387fa0512a0444f26abb8f63d8d2a43d5401da824a8f70433addb2f82e388: Status 404 returned error can't find the container with id dc2387fa0512a0444f26abb8f63d8d2a43d5401da824a8f70433addb2f82e388
Jul 17 11:54:09.014792 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:09.014706    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-zt729" event={"ID":"a8639e56-1824-4140-ace7-402d112c87ed","Type":"ContainerStarted","Data":"dc2387fa0512a0444f26abb8f63d8d2a43d5401da824a8f70433addb2f82e388"}
Jul 17 11:54:10.021742 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:10.021707    2563 generic.go:356] "Generic (PLEG): container finished" podID="a8639e56-1824-4140-ace7-402d112c87ed" containerID="9fd85816311266a0ce8b513fc07e5a1ee8ee98421b399df77ba533e45e3fce38" exitCode=6
Jul 17 11:54:10.022256 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:10.021761    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-zt729" event={"ID":"a8639e56-1824-4140-ace7-402d112c87ed","Type":"ContainerDied","Data":"9fd85816311266a0ce8b513fc07e5a1ee8ee98421b399df77ba533e45e3fce38"}
Jul 17 11:54:10.022256 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:10.022051    2563 scope.go:117] "RemoveContainer" containerID="9fd85816311266a0ce8b513fc07e5a1ee8ee98421b399df77ba533e45e3fce38"
Jul 17 11:54:10.237198 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:10.237162    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:54:10.461177 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:10.461143    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:54:10.915707 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:10.915551    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c9d884449-brg67_02fad526-849d-4a1e-a418-af6f54952872/authorino/0.log"
Jul 17 11:54:11.028493 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:11.028403    2563 generic.go:356] "Generic (PLEG): container finished" podID="a8639e56-1824-4140-ace7-402d112c87ed" containerID="be30e4f6a56ca2ae167c6f31251a4949f6646b292d4e59273e6d4a17eca53312" exitCode=6
Jul 17 11:54:11.028888 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:11.028487    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-zt729" event={"ID":"a8639e56-1824-4140-ace7-402d112c87ed","Type":"ContainerDied","Data":"be30e4f6a56ca2ae167c6f31251a4949f6646b292d4e59273e6d4a17eca53312"}
Jul 17 11:54:11.028888 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:11.028568    2563 scope.go:117] "RemoveContainer" containerID="9fd85816311266a0ce8b513fc07e5a1ee8ee98421b399df77ba533e45e3fce38"
Jul 17 11:54:11.028888 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:11.028693    2563 scope.go:117] "RemoveContainer" containerID="be30e4f6a56ca2ae167c6f31251a4949f6646b292d4e59273e6d4a17eca53312"
Jul 17 11:54:11.029013 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:54:11.028924    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45989be5-zt729_odh-ai-gateway-infra(a8639e56-1824-4140-ace7-402d112c87ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-zt729" podUID="a8639e56-1824-4140-ace7-402d112c87ed"
Jul 17 11:54:12.034828 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:12.034800    2563 scope.go:117] "RemoveContainer" containerID="be30e4f6a56ca2ae167c6f31251a4949f6646b292d4e59273e6d4a17eca53312"
Jul 17 11:54:12.035210 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:54:12.035017    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45989be5-zt729_odh-ai-gateway-infra(a8639e56-1824-4140-ace7-402d112c87ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-zt729" podUID="a8639e56-1824-4140-ace7-402d112c87ed"
Jul 17 11:54:17.089783 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:17.089696    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rc9bz_4ced4cbc-2492-41cd-b739-c8a54bade76b/manager/0.log"
Jul 17 11:54:17.213527 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:17.213500    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cd97f7fc7-54tcg_22f3c1ce-3171-49cf-8a07-4c0fcc2ae585/copy-manifests/0.log"
Jul 17 11:54:17.222200 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:17.222173    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cd97f7fc7-54tcg_22f3c1ce-3171-49cf-8a07-4c0fcc2ae585/manager/0.log"
Jul 17 11:54:18.327326 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:18.327299    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-45989be5-zt729_a8639e56-1824-4140-ace7-402d112c87ed/revoke-keys/1.log"
Jul 17 11:54:18.439383 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:18.439356    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-c0bf60-mfb52_bd7a63a9-2208-46a6-99d6-1251e3a4ad17/revoke-keys/0.log"
Jul 17 11:54:18.554312 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:18.554282    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-a50d5b-gxsxr_b2004a40-8448-48de-b065-3c5a61bf589c/revoke-keys/0.log"
Jul 17 11:54:19.028307 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:19.028277    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-63359af2-7gpnl_51fb0c2f-1075-47ee-93ce-5dc80356aab5/revoke-keys/0.log"
Jul 17 11:54:20.190888 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:20.190854    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb_7e09d1b3-7cdb-44b1-a080-5381ac1db7d5/util/0.log"
Jul 17 11:54:20.198783 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:20.198753    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb_7e09d1b3-7cdb-44b1-a080-5381ac1db7d5/pull/0.log"
Jul 17 11:54:20.205994 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:20.205969    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb_7e09d1b3-7cdb-44b1-a080-5381ac1db7d5/extract/0.log"
Jul 17 11:54:20.322344 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:20.322309    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl_f38ed71a-8a3c-44b6-9564-78ae054df646/pull/0.log"
Jul 17 11:54:20.330652 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:20.330630    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl_f38ed71a-8a3c-44b6-9564-78ae054df646/extract/0.log"
Jul 17 11:54:20.339271 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:20.339249    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl_f38ed71a-8a3c-44b6-9564-78ae054df646/util/0.log"
Jul 17 11:54:20.454314 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:20.454236    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls_d954e900-5ec9-46df-8d79-eaa33f8fdf5b/util/0.log"
Jul 17 11:54:20.461285 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:20.461262    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls_d954e900-5ec9-46df-8d79-eaa33f8fdf5b/pull/0.log"
Jul 17 11:54:20.468164 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:20.468147    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls_d954e900-5ec9-46df-8d79-eaa33f8fdf5b/extract/0.log"
Jul 17 11:54:20.589216 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:20.589189    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh_229ead90-eb4a-4763-b2bd-757a963dea6b/util/0.log"
Jul 17 11:54:20.596513 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:20.596491    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh_229ead90-eb4a-4763-b2bd-757a963dea6b/pull/0.log"
Jul 17 11:54:20.603698 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:20.603679    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh_229ead90-eb4a-4763-b2bd-757a963dea6b/extract/0.log"
Jul 17 11:54:20.734863 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:20.734768    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c9d884449-brg67_02fad526-849d-4a1e-a418-af6f54952872/authorino/0.log"
Jul 17 11:54:21.952234 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:21.952207    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4l7mvw_d6c6f4f1-bf17-4be7-b422-99093e48f316/istio-proxy/0.log"
Jul 17 11:54:22.311786 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:22.311701    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c556c8c74-kh79v_2a51e381-c349-4c91-b00c-460e28017f4a/kube-auth-proxy/0.log"
Jul 17 11:54:23.530674 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:23.530641    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbjzbzs_28fbea3e-f044-4134-a38d-bf84c02cb151/main/0.log"
Jul 17 11:54:23.648175 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:23.648145    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-t7jtv_e6995c1c-928c-415e-9353-99d9ed856ecf/main/0.log"
Jul 17 11:54:23.770664 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:23.770632    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-pkpq7_51f661df-6c4f-47a8-8bff-244b58486a70/main/0.log"
Jul 17 11:54:25.662745 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:25.662720    2563 scope.go:117] "RemoveContainer" containerID="be30e4f6a56ca2ae167c6f31251a4949f6646b292d4e59273e6d4a17eca53312"
Jul 17 11:54:26.108690 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:26.108661    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-zt729" event={"ID":"a8639e56-1824-4140-ace7-402d112c87ed","Type":"ContainerStarted","Data":"43794ea47bb0d0b2c3b8b7f64a064454a72b07e955fc747c490ee58cd9368e96"}
Jul 17 11:54:26.132839 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:26.132795    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-zt729" podStartSLOduration=18.13278245 podStartE2EDuration="18.13278245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 11:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:54:26.132403023 +0000 UTC m=+3421.061942663" watchObservedRunningTime="2026-07-17 11:54:26.13278245 +0000 UTC m=+3421.062322136"
Jul 17 11:54:27.115372 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:27.115341    2563 generic.go:356] "Generic (PLEG): container finished" podID="a8639e56-1824-4140-ace7-402d112c87ed" containerID="43794ea47bb0d0b2c3b8b7f64a064454a72b07e955fc747c490ee58cd9368e96" exitCode=6
Jul 17 11:54:27.115780 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:27.115412    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-zt729" event={"ID":"a8639e56-1824-4140-ace7-402d112c87ed","Type":"ContainerDied","Data":"43794ea47bb0d0b2c3b8b7f64a064454a72b07e955fc747c490ee58cd9368e96"}
Jul 17 11:54:27.115780 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:27.115468    2563 scope.go:117] "RemoveContainer" containerID="be30e4f6a56ca2ae167c6f31251a4949f6646b292d4e59273e6d4a17eca53312"
Jul 17 11:54:27.115780 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:27.115694    2563 scope.go:117] "RemoveContainer" containerID="43794ea47bb0d0b2c3b8b7f64a064454a72b07e955fc747c490ee58cd9368e96"
Jul 17 11:54:27.115913 ip-10-0-140-233 kubenswrapper[2563]: E0717 11:54:27.115887    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45989be5-zt729_odh-ai-gateway-infra(a8639e56-1824-4140-ace7-402d112c87ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-zt729" podUID="a8639e56-1824-4140-ace7-402d112c87ed"
Jul 17 11:54:27.149651 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:27.149625    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-zt729"]
Jul 17 11:54:28.253313 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:28.253292    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-zt729"
Jul 17 11:54:28.293242 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:28.293215    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8639e56-1824-4140-ace7-402d112c87ed-maas-api-service-ca\") pod \"a8639e56-1824-4140-ace7-402d112c87ed\" (UID: \"a8639e56-1824-4140-ace7-402d112c87ed\") "
Jul 17 11:54:28.293530 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:28.293504    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8639e56-1824-4140-ace7-402d112c87ed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a8639e56-1824-4140-ace7-402d112c87ed" (UID: "a8639e56-1824-4140-ace7-402d112c87ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 11:54:28.394802 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:28.394771    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8639e56-1824-4140-ace7-402d112c87ed-maas-api-service-ca\") on node \"ip-10-0-140-233.ec2.internal\" DevicePath \"\""
Jul 17 11:54:29.129139 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:29.129111    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-zt729"
Jul 17 11:54:29.129339 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:29.129101    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-zt729" event={"ID":"a8639e56-1824-4140-ace7-402d112c87ed","Type":"ContainerDied","Data":"dc2387fa0512a0444f26abb8f63d8d2a43d5401da824a8f70433addb2f82e388"}
Jul 17 11:54:29.129339 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:29.129248    2563 scope.go:117] "RemoveContainer" containerID="43794ea47bb0d0b2c3b8b7f64a064454a72b07e955fc747c490ee58cd9368e96"
Jul 17 11:54:29.153966 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:29.153944    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-zt729"]
Jul 17 11:54:29.158728 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:29.158707    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45989be5-zt729"]
Jul 17 11:54:29.666562 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:29.666531    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8639e56-1824-4140-ace7-402d112c87ed" path="/var/lib/kubelet/pods/a8639e56-1824-4140-ace7-402d112c87ed/volumes"
Jul 17 11:54:30.010483 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:30.010404    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:54:30.112840 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:30.112807    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log" limit=1024
Jul 17 11:54:33.052063 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:33.052035    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-b7fx4_84ae7bd0-6126-491f-9126-2d64506af889/global-pull-secret-syncer/0.log"
Jul 17 11:54:33.116261 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:33.116229    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kz6gd_73402c16-36b4-442c-9009-4d41bd9003fc/konnectivity-agent/0.log"
Jul 17 11:54:33.269360 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:33.269335    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-233.ec2.internal_098d9536b1eb81f9184c22a2bb857164/haproxy/0.log"
Jul 17 11:54:37.427152 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:37.427116    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb_7e09d1b3-7cdb-44b1-a080-5381ac1db7d5/extract/0.log"
Jul 17 11:54:37.454260 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:37.454225    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb_7e09d1b3-7cdb-44b1-a080-5381ac1db7d5/util/0.log"
Jul 17 11:54:37.480523 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:37.480495    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dq2wb_7e09d1b3-7cdb-44b1-a080-5381ac1db7d5/pull/0.log"
Jul 17 11:54:37.528765 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:37.528722    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl_f38ed71a-8a3c-44b6-9564-78ae054df646/extract/0.log"
Jul 17 11:54:37.556180 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:37.556154    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl_f38ed71a-8a3c-44b6-9564-78ae054df646/util/0.log"
Jul 17 11:54:37.582472 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:37.582444    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0r86dl_f38ed71a-8a3c-44b6-9564-78ae054df646/pull/0.log"
Jul 17 11:54:37.611837 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:37.611818    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls_d954e900-5ec9-46df-8d79-eaa33f8fdf5b/extract/0.log"
Jul 17 11:54:37.635787 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:37.635765    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls_d954e900-5ec9-46df-8d79-eaa33f8fdf5b/util/0.log"
Jul 17 11:54:37.660486 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:37.660467    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xv4ls_d954e900-5ec9-46df-8d79-eaa33f8fdf5b/pull/0.log"
Jul 17 11:54:37.691103 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:37.691032    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh_229ead90-eb4a-4763-b2bd-757a963dea6b/extract/0.log"
Jul 17 11:54:37.715059 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:37.715036    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh_229ead90-eb4a-4763-b2bd-757a963dea6b/util/0.log"
Jul 17 11:54:37.740476 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:37.740447    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1klnbh_229ead90-eb4a-4763-b2bd-757a963dea6b/pull/0.log"
Jul 17 11:54:38.239173 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:38.239126    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c9d884449-brg67_02fad526-849d-4a1e-a418-af6f54952872/authorino/0.log"
Jul 17 11:54:40.647200 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:40.647170    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-5xw7l_f8f5769f-44b8-49e1-acc9-35fc5b558035/cluster-monitoring-operator/0.log"
Jul 17 11:54:40.814837 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:40.814810    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7f4c698489-2952v_351ba54a-209f-4ec6-8f6a-56bbef8f3e68/metrics-server/0.log"
Jul 17 11:54:40.986256 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:40.986175    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gs7hc_c8e6f422-93f9-49cb-8ff3-cad07c26c9e4/node-exporter/0.log"
Jul 17 11:54:41.013778 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:41.013754    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gs7hc_c8e6f422-93f9-49cb-8ff3-cad07c26c9e4/kube-rbac-proxy/0.log"
Jul 17 11:54:41.036719 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:41.036700    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gs7hc_c8e6f422-93f9-49cb-8ff3-cad07c26c9e4/init-textfile/0.log"
Jul 17 11:54:41.164721 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:41.164695    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-gmdhf_e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0/kube-rbac-proxy-main/0.log"
Jul 17 11:54:41.191254 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:41.191232    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-gmdhf_e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0/kube-rbac-proxy-self/0.log"
Jul 17 11:54:41.221795 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:41.221775    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-gmdhf_e0dd40d2-dfb1-416d-8bbf-18a1f2dbc8a0/openshift-state-metrics/0.log"
Jul 17 11:54:41.643877 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:41.643847    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78cd79d867-5lv8s_f440b569-a68f-48e6-8a50-51b48d4e71e1/thanos-query/0.log"
Jul 17 11:54:41.669139 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:41.669113    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78cd79d867-5lv8s_f440b569-a68f-48e6-8a50-51b48d4e71e1/kube-rbac-proxy-web/0.log"
Jul 17 11:54:41.698184 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:41.698157    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78cd79d867-5lv8s_f440b569-a68f-48e6-8a50-51b48d4e71e1/kube-rbac-proxy/0.log"
Jul 17 11:54:41.724909 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:41.724881    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78cd79d867-5lv8s_f440b569-a68f-48e6-8a50-51b48d4e71e1/prom-label-proxy/0.log"
Jul 17 11:54:41.750902 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:41.750882    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78cd79d867-5lv8s_f440b569-a68f-48e6-8a50-51b48d4e71e1/kube-rbac-proxy-rules/0.log"
Jul 17 11:54:41.778545 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:41.778520    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78cd79d867-5lv8s_f440b569-a68f-48e6-8a50-51b48d4e71e1/kube-rbac-proxy-metrics/0.log"
Jul 17 11:54:43.441283 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.441246    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lgfmw/perf-node-gather-daemonset-mpz7d"]
Jul 17 11:54:43.441655 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.441613    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c1b08e5-60d6-4fdc-83eb-d5365a9d0661" containerName="istio-proxy"
Jul 17 11:54:43.441655 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.441624    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c1b08e5-60d6-4fdc-83eb-d5365a9d0661" containerName="istio-proxy"
Jul 17 11:54:43.441655 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.441635    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8639e56-1824-4140-ace7-402d112c87ed" containerName="revoke-keys"
Jul 17 11:54:43.441655 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.441640    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8639e56-1824-4140-ace7-402d112c87ed" containerName="revoke-keys"
Jul 17 11:54:43.441795 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.441668    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8639e56-1824-4140-ace7-402d112c87ed" containerName="revoke-keys"
Jul 17 11:54:43.441795 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.441674    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8639e56-1824-4140-ace7-402d112c87ed" containerName="revoke-keys"
Jul 17 11:54:43.441795 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.441721    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8639e56-1824-4140-ace7-402d112c87ed" containerName="revoke-keys"
Jul 17 11:54:43.441795 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.441733    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8639e56-1824-4140-ace7-402d112c87ed" containerName="revoke-keys"
Jul 17 11:54:43.444774 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.444759    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lgfmw/perf-node-gather-daemonset-mpz7d"
Jul 17 11:54:43.447382 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.447362    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lgfmw\"/\"openshift-service-ca.crt\""
Jul 17 11:54:43.448481 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.448454    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lgfmw\"/\"default-dockercfg-tk7jl\""
Jul 17 11:54:43.448583 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.448490    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lgfmw\"/\"kube-root-ca.crt\""
Jul 17 11:54:43.454137 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.454117    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lgfmw/perf-node-gather-daemonset-mpz7d"]
Jul 17 11:54:43.524880 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.524854    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/424717f8-39a3-41d6-931d-11c678148ce9-proc\") pod \"perf-node-gather-daemonset-mpz7d\" (UID: \"424717f8-39a3-41d6-931d-11c678148ce9\") " pod="openshift-must-gather-lgfmw/perf-node-gather-daemonset-mpz7d"
Jul 17 11:54:43.525021 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.524887    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7sw2\" (UniqueName: \"kubernetes.io/projected/424717f8-39a3-41d6-931d-11c678148ce9-kube-api-access-j7sw2\") pod \"perf-node-gather-daemonset-mpz7d\" (UID: \"424717f8-39a3-41d6-931d-11c678148ce9\") " pod="openshift-must-gather-lgfmw/perf-node-gather-daemonset-mpz7d"
Jul 17 11:54:43.525021 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.524935    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/424717f8-39a3-41d6-931d-11c678148ce9-podres\") pod \"perf-node-gather-daemonset-mpz7d\" (UID: \"424717f8-39a3-41d6-931d-11c678148ce9\") " pod="openshift-must-gather-lgfmw/perf-node-gather-daemonset-mpz7d"
Jul 17 11:54:43.525021 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.524978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/424717f8-39a3-41d6-931d-11c678148ce9-sys\") pod \"perf-node-gather-daemonset-mpz7d\" (UID: \"424717f8-39a3-41d6-931d-11c678148ce9\") " pod="openshift-must-gather-lgfmw/perf-node-gather-daemonset-mpz7d"
Jul 17 11:54:43.525222 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.525031    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/424717f8-39a3-41d6-931d-11c678148ce9-lib-modules\") pod \"perf-node-gather-daemonset-mpz7d\" (UID: \"424717f8-39a3-41d6-931d-11c678148ce9\") " pod="openshift-must-gather-lgfmw/perf-node-gather-daemonset-mpz7d"
Jul 17 11:54:43.626012 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.625985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/424717f8-39a3-41d6-931d-11c678148ce9-podres\") pod \"perf-node-gather-daemonset-mpz7d\" (UID: \"424717f8-39a3-41d6-931d-11c678148ce9\") " pod="openshift-must-gather-lgfmw/perf-node-gather-daemonset-mpz7d"
Jul 17 11:54:43.626127 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.626015    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/424717f8-39a3-41d6-931d-11c678148ce9-sys\") pod \"perf-node-gather-daemonset-mpz7d\" (UID: \"424717f8-39a3-41d6-931d-11c678148ce9\") " pod="openshift-must-gather-lgfmw/perf-node-gather-daemonset-mpz7d"
Jul 17 11:54:43.626127 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.626032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/424717f8-39a3-41d6-931d-11c678148ce9-lib-modules\") pod \"perf-node-gather-daemonset-mpz7d\" (UID: \"424717f8-39a3-41d6-931d-11c678148ce9\") " pod="openshift-must-gather-lgfmw/perf-node-gather-daemonset-mpz7d"
Jul 17 11:54:43.626258 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.626131    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/424717f8-39a3-41d6-931d-11c678148ce9-sys\") pod \"perf-node-gather-daemonset-mpz7d\" (UID: \"424717f8-39a3-41d6-931d-11c678148ce9\") " pod="openshift-must-gather-lgfmw/perf-node-gather-daemonset-mpz7d"
Jul 17 11:54:43.626258 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.626167    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/424717f8-39a3-41d6-931d-11c678148ce9-lib-modules\") pod \"perf-node-gather-daemonset-mpz7d\" (UID: \"424717f8-39a3-41d6-931d-11c678148ce9\") " pod="openshift-must-gather-lgfmw/perf-node-gather-daemonset-mpz7d"
Jul 17 11:54:43.626258 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.626170    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/424717f8-39a3-41d6-931d-11c678148ce9-podres\") pod \"perf-node-gather-daemonset-mpz7d\" (UID: \"424717f8-39a3-41d6-931d-11c678148ce9\") " pod="openshift-must-gather-lgfmw/perf-node-gather-daemonset-mpz7d"
Jul 17 11:54:43.626258 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.626188    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/424717f8-39a3-41d6-931d-11c678148ce9-proc\") pod \"perf-node-gather-daemonset-mpz7d\" (UID: \"424717f8-39a3-41d6-931d-11c678148ce9\") " pod="openshift-must-gather-lgfmw/perf-node-gather-daemonset-mpz7d"
Jul 17 11:54:43.626258 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.626230    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7sw2\" (UniqueName: \"kubernetes.io/projected/424717f8-39a3-41d6-931d-11c678148ce9-kube-api-access-j7sw2\") pod \"perf-node-gather-daemonset-mpz7d\" (UID: \"424717f8-39a3-41d6-931d-11c678148ce9\") " pod="openshift-must-gather-lgfmw/perf-node-gather-daemonset-mpz7d"
Jul 17 11:54:43.626450 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.626278    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/424717f8-39a3-41d6-931d-11c678148ce9-proc\") pod \"perf-node-gather-daemonset-mpz7d\" (UID: \"424717f8-39a3-41d6-931d-11c678148ce9\") " pod="openshift-must-gather-lgfmw/perf-node-gather-daemonset-mpz7d"
Jul 17 11:54:43.635525 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.635497    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7sw2\" (UniqueName: \"kubernetes.io/projected/424717f8-39a3-41d6-931d-11c678148ce9-kube-api-access-j7sw2\") pod \"perf-node-gather-daemonset-mpz7d\" (UID: \"424717f8-39a3-41d6-931d-11c678148ce9\") " pod="openshift-must-gather-lgfmw/perf-node-gather-daemonset-mpz7d"
Jul 17 11:54:43.756419 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.756346    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lgfmw/perf-node-gather-daemonset-mpz7d"
Jul 17 11:54:43.887852 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:43.887830    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lgfmw/perf-node-gather-daemonset-mpz7d"]
Jul 17 11:54:43.889412 ip-10-0-140-233 kubenswrapper[2563]: W0717 11:54:43.889385    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod424717f8_39a3_41d6_931d_11c678148ce9.slice/crio-4b26f9f048e1281519d131bc0c79e465cf0d0ab3d4244db3da827786cb628cd1 WatchSource:0}: Error finding container 4b26f9f048e1281519d131bc0c79e465cf0d0ab3d4244db3da827786cb628cd1: Status 404 returned error can't find the container with id 4b26f9f048e1281519d131bc0c79e465cf0d0ab3d4244db3da827786cb628cd1
Jul 17 11:54:44.212921 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:44.212885    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lgfmw/perf-node-gather-daemonset-mpz7d" event={"ID":"424717f8-39a3-41d6-931d-11c678148ce9","Type":"ContainerStarted","Data":"5b7fa92ef53e8730e0e4e2af3178d8ea181fb5c39aca087b3ad75a37865bee7c"}
Jul 17 11:54:44.213160 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:44.212953    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lgfmw/perf-node-gather-daemonset-mpz7d"
Jul 17 11:54:44.213160 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:44.212973    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lgfmw/perf-node-gather-daemonset-mpz7d" event={"ID":"424717f8-39a3-41d6-931d-11c678148ce9","Type":"ContainerStarted","Data":"4b26f9f048e1281519d131bc0c79e465cf0d0ab3d4244db3da827786cb628cd1"}
Jul 17 11:54:44.232050 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:44.231995    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lgfmw/perf-node-gather-daemonset-mpz7d" podStartSLOduration=1.231980188 podStartE2EDuration="1.231980188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 11:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 11:54:44.230033042 +0000 UTC m=+3439.159572683" watchObservedRunningTime="2026-07-17 11:54:44.231980188 +0000 UTC m=+3439.161519829"
Jul 17 11:54:46.322163 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:46.322126    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h4cnn_bef28d6a-f451-4c22-adca-7179590804ea/dns/0.log"
Jul 17 11:54:46.347698 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:46.347676    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h4cnn_bef28d6a-f451-4c22-adca-7179590804ea/kube-rbac-proxy/0.log"
Jul 17 11:54:46.402094 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:46.402057    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-87r57_f1f509b8-7e29-445b-ad25-dd54a1c6dc07/dns-node-resolver/0.log"
Jul 17 11:54:47.020834 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:47.020806    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7fc569f596-rgpxb_632083f5-6ec0-4455-8b41-f372f69294fc/registry/0.log"
Jul 17 11:54:47.113344 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:47.113309    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qqqf9_72f5631e-c70d-47b8-9405-d8b986fd12e7/node-ca/0.log"
Jul 17 11:54:48.149443 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:48.149416    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4l7mvw_d6c6f4f1-bf17-4be7-b422-99093e48f316/istio-proxy/0.log"
Jul 17 11:54:48.320344 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:48.320317    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c556c8c74-kh79v_2a51e381-c349-4c91-b00c-460e28017f4a/kube-auth-proxy/0.log"
Jul 17 11:54:49.318977 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:49.318945    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gzhwh_eced893c-6854-4d8c-bbc2-6d6cfa602028/serve-healthcheck-canary/0.log"
Jul 17 11:54:49.940697 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:49.940666    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gf598_bf372042-c2bc-45b7-a95d-8a90116a13a7/kube-rbac-proxy/0.log"
Jul 17 11:54:49.967984 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:49.967963    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gf598_bf372042-c2bc-45b7-a95d-8a90116a13a7/exporter/0.log"
Jul 17 11:54:49.993093 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:49.993056    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gf598_bf372042-c2bc-45b7-a95d-8a90116a13a7/extractor/0.log"
Jul 17 11:54:50.227964 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:50.227880    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lgfmw/perf-node-gather-daemonset-mpz7d"
Jul 17 11:54:52.839495 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:52.839457    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rc9bz_4ced4cbc-2492-41cd-b739-c8a54bade76b/manager/0.log"
Jul 17 11:54:52.868346 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:52.868318    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cd97f7fc7-54tcg_22f3c1ce-3171-49cf-8a07-4c0fcc2ae585/manager/0.log"
Jul 17 11:54:52.895758 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:52.895736    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cd97f7fc7-54tcg_22f3c1ce-3171-49cf-8a07-4c0fcc2ae585/copy-manifests/0.log"
Jul 17 11:54:54.498872 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:54.498846    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-d4bfb48f4-wdkwr_ef1e11f2-4d04-426e-b5b7-7ece736a18f8/manager/0.log"
Jul 17 11:54:54.525674 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:54.525646    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-zz7jl_3918ecd3-a833-43da-988f-df80290a9407/openshift-lws-operator/0.log"
Jul 17 11:54:54.635444 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:54.635415    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4l7mvw_d6c6f4f1-bf17-4be7-b422-99093e48f316/istio-proxy/0.log"
Jul 17 11:54:54.781351 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:54.781266    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c556c8c74-kh79v_2a51e381-c349-4c91-b00c-460e28017f4a/kube-auth-proxy/0.log"
Jul 17 11:54:56.177780 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:56.177744    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4l7mvw_d6c6f4f1-bf17-4be7-b422-99093e48f316/istio-proxy/0.log"
Jul 17 11:54:56.325194 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:54:56.325167    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c556c8c74-kh79v_2a51e381-c349-4c91-b00c-460e28017f4a/kube-auth-proxy/0.log"
Jul 17 11:55:00.326110 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:55:00.326057    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-h94fh_7cd8f40f-2f2d-47e1-b464-33828197574b/kube-storage-version-migrator-operator/1.log"
Jul 17 11:55:00.327473 ip-10-0-140-233 kubenswrapper[2563]: I0717 11:55:00.327446    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-h94fh_7cd8f40f-2f2d-47e1-b464-33828197574b/kube-storage-version-migrator-operator/0.log"